Kernel error kernel:[360783.353677]

Hi

Having upgraded earlier this week from V6, I just received the following kernel error:

2012/09/27 20:45:21 kernel:[360783.353677] Pid: 0, comm: swapper Not tainted 2.6.35.14-oe64-00000-ga36704d #69
2012/09/27 20:45:21 kernel:[360783.353679] Call Trace:
2012/09/27 20:45:21 kernel:[360783.353680] <IRQ> [<ffffffff814d3455>] ? dev_watchdog+0x215/0x220
2012/09/27 20:45:21 kernel:[360783.353684] [<ffffffff814d3455>] ? dev_watchdog+0x215/0x220
2012/09/27 20:45:21 kernel:[360783.353688] [<ffffffff8104590c>] ? warn_slowpath_common+0x8c/0xc0
2012/09/27 20:45:21 kernel:[360783.353691] [<ffffffff81045a16>] ? warn_slowpath_fmt+0x56/0x60
2012/09/27 20:45:21 kernel:[360783.353693] [<ffffffff8103cdf1>] ? enqueue_task+0x61/0xa0
2012/09/27 20:45:21 kernel:[360783.353695] [<ffffffff8103d873>] ? try_to_wake_up+0xb3/0x2b0
2012/09/27 20:45:21 kernel:[360783.353699] [<ffffffff8129091e>] ? strlcpy+0x4e/0x80
2012/09/27 20:45:21 kernel:[360783.353701] [<ffffffff814b9933>] ? netdev_drivername+0x43/0x50
2012/09/27 20:45:21 kernel:[360783.353703] [<ffffffff814d3455>] ? dev_watchdog+0x215/0x220
2012/09/27 20:45:21 kernel:[360783.353706] [<ffffffff81040d93>] ? __wake_up+0x43/0x70
2012/09/27 20:45:21 kernel:[360783.353708] [<ffffffff814d3240>] ? dev_watchdog+0x0/0x220
2012/09/27 20:45:21 kernel:[360783.353710] [<ffffffff8105122f>] ? run_timer_softirq+0x19f/0x200
2012/09/27 20:45:21 kernel:[360783.353712] [<ffffffff810649c2>] ? ktime_get+0x52/0xd0
2012/09/27 20:45:21 kernel:[360783.353715] [<ffffffff8104b5fd>] ? __do_softirq+0xad/0x150
2012/09/27 20:45:21 kernel:[360783.353718] [<ffffffff819e2140>] ? early_idt_handler+0x0/0x71
2012/09/27 20:45:21 kernel:[360783.353721] [<ffffffff8100ab5c>] ? call_softirq+0x1c/0x30
2012/09/27 20:45:21 kernel:[360783.353723] [<ffffffff8100c505>] ? do_softirq+0x65/0xa0
2012/09/27 20:45:21 kernel:[360783.353725] [<ffffffff8104b54c>] ? irq_exit+0x7c/0x80
2012/09/27 20:45:21 kernel:[360783.353728] [<ffffffff8102289a>] ? smp_apic_timer_interrupt+0x6a/0xa0
2012/09/27 20:45:21 kernel:[360783.353730] [<ffffffff8100a613>] ? apic_timer_interrupt+0x13/0x20
2012/09/27 20:45:21 kernel:[360783.353731] <EOI> [<ffffffffa00a0d6a>] ? acpi_idle_enter_simple+0xf7/0x128 [processor]
2012/09/27 20:45:21 kernel:[360783.353738] [<ffffffffa00a0d63>] ? acpi_idle_enter_simple+0xf0/0x128 [processor]
2012/09/27 20:45:21 kernel:[360783.353741] [<ffffffff81474298>] ? cpuidle_idle_call+0x98/0x110
2012/09/27 20:45:21 kernel:[360783.353744] [<ffffffff81008ba7>] ? cpu_idle+0x57/0x90
2012/09/27 20:45:21 kernel:[360783.353746] [<ffffffff819e2e25>] ? start_kernel+0x2e5/0x3c0
2012/09/27 20:45:21 kernel:[360783.353748] [<ffffffff819e23df>] ? x86_64_start_kernel+0xff/0x130

I don't know to what extent I should be worried, because everything seems to work just fine, but it might be useful for me and others to know how we should respond to such errors.

Could be an IRQ issue, try to disable any and all power management features from the BIOS of the motherboard. Also if you have additional NIC like a quad port or dual port nic maybe move that to a different slot.

Ok, will try that at the next reboot and will keep this thread up to date.
It just seemed to come completely out of the dark now, so I'll see if I can find out what unusual things happened around that time.

Well, the issue might be a different one:

Last week, I replaced all hardware (motherboard, processor, memory, nics) except the Areca Raid Controller and the DOM, after I had seen a apparently random few system freezes.
So far so good for last week, but half an hour ago, yet again, the system froze completely.

The RAID controller does not indicate any errors, so it might be the DOM (Transcend Industrial Flash 2GB SLC DOM with 1 million hours MTBF).
To be honest, I do not know what role the DOM plays once the server has started, but I thought that is minimal and all vital information is stored on the system volume.
Would it make sense to think it is unlikely the DOM is the problem here?

Anyway, this time I would like to know a little more precise what causes the problems, so I downloaded the log files but there seems to be no useful info in the critical_errors.log.
What other logs could reveal some useful hints what causes the freezes?

Try re-formatting the DOM after you save your settings in Maint. > Misc and take all options so that you can save the file called settings.cnf then use to restore. You can look at the dmesg2.log as well to see if there are any errors and the packages ones during the time of the crash.

Also try to "Disable" in BIOS all power management options.
Try to disable APM and ACPI in CTRL+ALT+T menu(in console mode ).ctrl+alt+t->boot options(9)->Boot parameters(1).

Ok, I turned off all power options in the BIOS.
Indeed it seemed to be the DOM: today the system froze again, and now it won't even boot beyond the "loading" screen.
So I unplugged the DOM and took an external USB drive and installed the software on that without any issue.

Unfortunately, after finishing the install and rebooting, it hangs on "loading" again, so I tried booting the old DOM in v6, but to no avail...
I had the RAID array checked during the night: zero errors.
But still: no DSS version is loading succesfully now.

So I took out all hardware and rebooted, et voila: it ran again.
I plugged all hardware in, changed the slots of the NICs to avoid any IRQ errors and booted succesfully again.
Apparently moving the NIC was helpful.

After that, I cleared the TDB Database and reconnected to the DC and got it running.
However, I faced once that the machine hang during booting (just after "init runlevel 2"), so I'll order a new DOM just to be sure...

Unfortunately, it is time for an update.

Having succesfully updated to v7, I am still experiencing system freezes that drive me nuts.

I looked in the dmesg.2 logfiles from the downloads I took after reboot, and in 2 out of 3 cases, no notification of any error was found.
The last time, a reference is made to the onboard SATA controller:

[ 663.417882] ata1: exception Emask 0x10 SAct 0x0 SErr 0x4000000 action 0xe frozen
[ 663.417885] ata1: irq_stat 0x00000040, connection status changed
[ 663.417887] ata1: SError: { DevExch }
[ 663.417893] ata1: hard resetting link
[ 664.165187] ata1: SATA link down (SStatus 0 SControl 300)
[ 664.185147] ata1: EH complete
[ 899.498772] ata1: exception Emask 0x10 SAct 0x0 SErr 0x4000000 action 0xe frozen
[ 899.498776] ata1: irq_stat 0x00000040, connection status changed
[ 899.498779] ata1: SError: { DevExch }
[ 899.498787] ata1: hard resetting link

However, no disk is connected to that port (only a single SSD on ata2), as can also be seen in earlier in the same logfile:

5.547281] NET: Registered protocol family 5
[ 5.638323] RPC: Registered rdma transport module.
[ 5.907834] ata1: SATA link down (SStatus 0 SControl 300)
[ 6.297160] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 6.298030] ata2.00: ACPI cmd ef/10:06:00:00:00:00 (SET FEATURES) succeeded
[ 6.298033] ata2.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out
[ 6.298035] ata2.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE CONFIGURATION OVERLAY) filtered out
[ 6.298566] ata2.00: ATA-9: OCZ-VERTEX4, 1.5, max UDMA/133
[ 6.298569] ata2.00: 500118192 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
[ 6.299425] ata2.00: ACPI cmd ef/10:06:00:00:00:00 (SET FEATURES) succeeded
[ 6.299427] ata2.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE LOCK) filtered out
[ 6.299429] ata2.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE CONFIGURATION OVERLAY) filtered out
[ 6.299968] ata2.00: configured for UDMA/133

After experiencing issues with another SSD (previously attached to ata1), I replaced almost all hardware and replaced the SSD with the one that is now attached to ata2.
The SSD is used as an iSCSI target (file i/o) and hosts a Windows 2012 VM (vSphere) with a small SQL 2012 database.

Strangely enough, I've ran this exact config for over 2 years without any problem, until 2 months ago.
Is there anyone that has a clue?

Did you do this:

Quote:

---

Try to disable APM and ACPI in CTRL+ALT+T menu(in console mode ).ctrl+alt+t->boot options(9)->Boot parameters(1).

---

I did, and I also disabled all ACPI functions in the machine's BIOS.
Furthermore, I've replaced the power supply yesterday and moved the SSD to the Areca Controller, to see if the onboard SATA controller might be the problem.

By the way -just for the information of others-, after moving the disk from the onboard controller to the Areca controller, another reboot was required before I could reconnect the Volume to the iSCSI target, but it worked surprisingly well.

Unfortunately, this morning the machine went down again, and not being home, I haven't had a chance to reset the machine and look at the logs.
Tonight, I'll bring the machine back up, check the ACPI settings, examine the log and post my findings here.

Are there any logs beside the dmesg.2 and critial_errors.log (so far: this was always empty) files that I should examine?

P.s. I found two other forum threads in which similar behavior is mentioned: here and here. With reference to the first, recalling when the freezes started, I think it was after upgrading above v6 update 90.
Unfortunately, in neither thread a final outcome is reported.

One of the links show a solution by replacing disk(s) with errors, after that the system start to work fine.
So we recommend you to check your controller health and to check the storage disks, then to run file system repair tool to fix your FS if it contain any issue.
Please check this KB: http://kb.open-e.com/File-system-repair_138.html

Also please be sure that your controller Firmware is updated to the latest firmware available at your controller website.

Thanks for the hints: I've checked the controller's and disks' health and no issue was found. All are running the latest firmware builds.
However, I found that the ACPI features were still turned on in the boot options - I guess I turned them off before upgrading to v7 and forgot to check afterwards.
I turned it off and rebooted.

The machine is up and running again and I keep my fingers crossed...
Thanks for your help and I'll keep this thread up to date so it might be helpful for other users (although maybe just the inexperienced users like me).

FYI: driver settings, and boot settings and a few more, are not saved when going from DSS V6-->DSS V7. I thought this was in the release notes someplace. Guess I will have to check on that!

I'm sure it's in there: I probably just did not pay enough attention when upgrading :)

Unfortunately, keeping the fingers crossed did not help: even with having ran a full memtest (no errors), completely disabled the onboard SATA controller and the ACPI features turned off, the system hang again last night.
I rebooted and checked the error logs, but I could not find a clue what happened.

It feels like I'm slowly running out of ideas...
Anyone willing to take a look at my logs and see if I overlooked something?

yeah, for sure.

How do you see that it is a system freeze? Is it a actual freeze? Or is it something like: WebGUI not available, Console behaving slow and strange?

Thanks for the offer: I have to find a way to post the latest Logs here (if that is permitted).

Well, the freeze is quite comprehensive: no response on NICs (not able to ping), no response on console (no keystroke wakes the screen) at all.
All that is left, is a hard reset.

hm okay this is definitely a kernel panic.
just upload the logs anywhere and post a link to it? should be permitted ;)

Hi Daniel

It looks indeed like a kernel panic, mostly happening under some load. I've checked the logs for the latest connection logs before (manual) reset, and here are the results:

26-10, 02:00: Freeze when scanning folders for music files, logs taken at 07:13:

Quote:

---

[2012/10/26 02:00:49.934090, 1] smbd/vfs.c:932(check_reduced_name)
check_reduced_name: couldn't get realpath for

---

24-10, 16:10: Freeze when re-applying security rights, logs taken at 16:50:

Quote:

---

[2012/10/24 04:00:39.437702, 1] smbd/vfs.c:932(check_reduced_name)
check_reduced_name: couldn't get realpath for

---

Quote:

---

[2012/10/24 16:10:18.187779, 0] modules/vfs_posixacl.c:349(smb_acl_to_posix):
smb_acl_to_posix: ACL is invalid for set (Invalid argument)

---

23-10, 00:21: Freeze, not clear what triggered the panic, logs taken at 07:40:

Quote:

---

[2012/10/23 00:01:35.570535, 1] smbd/vfs.c:932(check_reduced_name)
check_reduced_name: couldn't get realpath

---

Quote:

---

[2012/10/23 08:05:50.234214, 1] smbd/vfs.c:932(check_reduced_name)
check_reduced_name: couldn't get realpath

---

22-10, 15:06, Freeze, not clear what triggered the panic, logs taken at 21:54

Quote:

---

[2012/10/22 14:29:51.344227, 1] smbd/vfs.c:932(check_reduced_name)
check_reduced_name: couldn't get realpath for

---

Admittedly, when bringing the machine back up, I did not always run the filesystem check: there was simply not enough time for that.
Could any FS errors cause a kernel panic?

Hi,

normally not.

Please send us the following logs completely(!):

critical_error, dmesg2, test, Samba

Greets

I just posted you a message on your profile page that contains the details of where the log can be found.
Please let me know if you are successful in picking them up...

Strange. Do you have any spare memory for that machine?

I don't, but I could take out 1 module (4GB remaining), although I wonder: a full pass on memtest did not reveal any errors.

There are only 3 things remaining that I can think of:

1. Now that I recall: I disabled the APM and ACPI boot options, but two (?) other were still enabled. Would it be worth a try to turn them off?
2. Could it be a problem with the USB DOM that hosts the OS? I could try running on another USB device.
3. Would it be worth a try to run a previous version that ran stable for a long time, i.e. v6 update 90? Are there any known changes that could cause this random freeze?

P.s. I just saw by the inability to reach my website that the machine froze again, this time after only a few hours - it's slowly getting hopeless...

If we assume that the mainboard and the cpu is not faulty (which is very rare) and that all drivers are tested and your system is on the HCL of open-e there is only one thing left: the memory.
Please note that you need a minimum of 4 hours memtest to see if everything is okay.
Best would be if you test it much longer!

The motherboard and cpu have been replaced recently, and besides that, the system only contains a Transcend USB DOM, an Areca 1220 controller and two Intel NICs. As far as I know, these are all on the HCL and they've ran without any issues for over 2 years.

I will run a memtest during the night and see what that gives.

Well, the memtest ran for 7 hours, without any errors.
Nonetheless, I am getting more and more convinced that it is a hardware error, more than a software error.
Since the freeze happens mostly under a certain (known) load, I suspect the NICs.
I've changed the network configuration as a test, and will replace the NICs as soon as possible.

For the sake of shortening other users' problem cycles in the future, I'll keep this thread posted...

After a little short to a week, I'm back, unfortunately.
Last week, I changed the NIC configuration and brought the machine back up again.

It ran stable for a week, and I've seriously stressed the machine without issues.
Tonight, however, just when really nothing special was going on (machine was almost idle), I saw it happening again:
The SMB connections were cut off, followed by the iSCSI connections a few minutes later.
At first, a ping was still possible, but soon the machine froze completely, including the console.

So, again, I rebooted, downloaded the logs and went to see what happened. Here's what I found:

Quote:

---

[2012/11/02 21:47:01.820369, 1] smbd/service.c:1070(make_connection_snum)
192.168.47.11 (192.168.47.11) connect to service Music initially as user DOMAIN+USERNAME1 (uid=102, gid=107) (pid 32727)
[2012/11/02 21:47:13.160369, 1] smbd/service.c:1251(close_cnum)
192.168.47.11 (192.168.47.11) closed connection to service Music
[2012/11/02 21:57:01.840369, 0] lib/fault.c:46(fault_report)
================================================== =============
[2012/11/02 20:57:01.850369, 0] lib/fault.c:47(fault_report)
INTERNAL ERROR: Signal 7 in pid 32727 (3.5.4)
Please read the Trouble-Shooting section of the Samba3-HOWTO
[2012/11/02 20:57:01.850369, 0] lib/fault.c:49(fault_report)

From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf
[2012/11/02 20:57:01.850369, 0] lib/fault.c:50(fault_report)
================================================== =============
[2012/11/02 20:57:01.850369, 0] lib/util.c:1465(smb_panic)
PANIC (pid 32727): internal error
[2012/11/02 20:57:01.850369, 0] lib/util.c:1569(log_stack_trace)
BACKTRACE: 0 stack frames:
[2012/11/02 20:57:01.850369, 0] lib/fault.c:326(dump_core)
dumping core in /usr/local/samba/var/cores/smbd

---

After finding the connection was lost, I tried to reconnect a share:

Quote:

---

[2012/11/02 21:40:14.290369, 1] smbd/service.c:1070(make_connection_snum)
192.168.47.117 (192.168.47.117) connect to service Pictures initially as user DOMAIN+USERNAME2 (uid=0, gid=107) (pid 16420)
[2012/11/02 21:40:25.680369, 1] smbd/service.c:1251(close_cnum)
192.168.47.117 (192.168.47.117) closed connection to service Music
[2012/11/02 21:40:25.680369, 1] smbd/service.c:1251(close_cnum)
192.168.47.117 (192.168.47.117) closed connection to service Pictures
[2012/11/02 21:45:20.340369, 0] lib/fault.c:46(fault_report)
================================================== =============
[2012/11/02 21:45:20.340369, 0] lib/fault.c:47(fault_report)
INTERNAL ERROR: Signal 7 in pid 16420 (3.5.4)
Please read the Trouble-Shooting section of the Samba3-HOWTO
[2012/11/02 21:45:20.340369, 0] lib/fault.c:49(fault_report)

From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf
[2012/11/02 21:45:20.340369, 0] lib/fault.c:50(fault_report)
================================================== =============
[2012/11/02 21:45:20.340369, 0] lib/util.c:1465(smb_panic)
PANIC (pid 16420): internal error
[2012/11/02 21:45:20.340369, 0] lib/util.c:1569(log_stack_trace)
BACKTRACE: 0 stack frames:
[2012/11/02 21:45:20.350369, 0] lib/fault.c:326(dump_core)
dumping core in /usr/local/samba/var/cores/smbd
[2012/11/02 21:45:20.730369, 1] smbd/service.c:1070(make_connection_snum)
192.168.47.117 (192.168.47.117) connect to service Movies initially as user DOMAIN+USERNAME2 (uid=0, gid=107) (pid 9889)
[2012/11/02 21:45:45.880369, 0] lib/fault.c:46(fault_report)
================================================== =============

---

Not only looks the time registration in the first part a little weird, I cannot relate it to any process at the client with IP 192.168.47.11, any machine stress in general, nor any hardware issue.
Please help!

There is one last thing I can think of: VMWare's iSCSI initiator causes the Open-E network stack to freeze the system.
Possible cause: http://vmtoday.com/2012/02/vsphere-5...oftware-iscsi/.

I found this problem to be the case in my vSphere machine. With hindsight, the problem occurred first with ESXi 5.0...