nfsd Not tainted 2.6.25.13-oe32

[lorenzo.salvadorini]

We receive one or two mail messages a day from our DSS storage, that I can't understand:

The first say:

 2009/12/24 15:40:45     kernel:Pid: 10041, comm: nfsd Not tainted 2.6.25.13-oe32 #61

After a minute, the second mail:

2009/12/24 15:40:45     kernel:[<c0156fef>] __alloc_pages+0x2df/0x330

2009/12/24 15:40:45     kernel:[<c016f805>] kmem_getpages+0x45/0x110

2009/12/24 15:40:45     kernel:[<c0170394>] cache_grow+0x104/0x140

2009/12/24 15:40:45     kernel:[<c0170587>] cache_alloc_refill+0x1b7/0x1f0

2009/12/24 15:40:45     kernel:[<c0170921>] __kmalloc+0xb1/0xc0

2009/12/24 15:40:45     kernel:[<c0476873>] __alloc_skb+0x43/0x100

2009/12/24 15:40:45     kernel:[<c04a8ac1>] tcp_collapse+0x171/0x3e0

2009/12/24 15:40:45     kernel:[<c04a8ee5>] tcp_prune_queue+0x65/0x130

2009/12/24 15:40:45     kernel:[<c04a8610>] tcp_data_queue+0x490/0x7d0

2009/12/24 15:40:45     kernel:[<c04a97b7>] tcp_rcv_established+0x387/0x740

2009/12/24 15:40:45     kernel:[<c04b0771>] tcp_v4_do_rcv+0xc1/0xd0

2009/12/24 15:40:45     kernel:[<c04b0d0d>] tcp_v4_rcv+0x58d/0x610

2009/12/24 15:40:45     kernel:[<c0496077>] ip_local_deliver_finish+0x47/0x150

2009/12/24 15:40:45     kernel:[<c04961fd>] ip_local_deliver+0x7d/0x90

2009/12/24 15:40:45     kernel:[<c0496253>] ip_rcv_finish+0x43/0x2e0

2009/12/24 15:40:45     kernel:[<c049667d>] ip_rcv+0x18d/0x290

2009/12/24 15:40:45     kernel:[<c047db78>] netif_receive_skb+0x218/0x250

2009/12/24 15:40:45     kernel:[<c047dc1c>] process_backlog+0x6c/0xd0

2009/12/24 15:40:45     kernel:[<c047dd4a>] net_rx_action+0x7a/0x150

2009/12/24 15:40:45     kernel:[<c0126d68>] __do_softirq+0x78/0xf0

2009/12/24 15:40:45     kernel:[<c0126e18>] do_softirq+0x38/0x40

2009/12/24 15:40:45     kernel:[<c0126eed>] irq_exit+0x6d/0x80

2009/12/24 15:40:45     kernel:[<c0105b7d>] do_IRQ+0x3d/0x70

2009/12/24 15:40:45     kernel:[<c01049c3>] common_interrupt+0x23/0x30

2009/12/24 15:40:45     kernel:[<c01a007b>] create_entry+0x2b/0x350

2009/12/24 15:40:45     kernel:[<c02a67c1>] xfsbufd_wakeup+0x21/0x50

2009/12/24 15:40:45     kernel:[<c015ad9f>] shrink_slab+0x6f/0x1c0

2009/12/24 15:40:45     kernel:[<c015c39a>] do_try_to_free_pages+0xca/0x1c0

2009/12/24 15:40:45     kernel:[<c015c4f7>] try_to_free_pages+0x67/0x70

2009/12/24 15:40:45     kernel:[<c015b820>] isolate_pages_global+0x0/0x60

2009/12/24 15:40:45     kernel:[<c0156e3c>] __alloc_pages+0x12c/0x330

2009/12/24 15:40:45     kernel:[<c04f68d1>] svc_recv+0xd1/0x4f0

2009/12/24 15:40:45     kernel:[<c011dd30>] default_wake_function+0x0/0x10

2009/12/24 15:40:45     kernel:[<c0215e50>] nfs3svc_encode_attrstat+0x0/0x80

2009/12/24 15:40:45     kernel:[<c04ed89d>] svc_process+0x46d/0x6e0

2009/12/24 15:40:45     kernel:[<c011dd30>] default_wake_function+0x0/0x10

2009/12/24 15:40:45     kernel:[<c012b79b>] recalc_sigpending+0xb/0x20

2009/12/24 15:40:45     kernel:[<c0208ecc>] nfsd+0xcc/0x2a0

2009/12/24 15:40:45     kernel:[<c0208e00>] nfsd+0x0/0x2a0

2009/12/24 15:40:45     kernel:[<c0104c47>] kernel_thread_helper+0x7/0x10

2009/12/24 15:40:45     kernel:=======================

2009/12/24 15:40:45     kernel:DMA per-cpu:

2009/12/24 15:40:45     kernel:CPU    0: hi:    0, btch:   1 usd:   0

2009/12/24 15:40:45     kernel:CPU    1: hi:    0, btch:   1 usd:   0

2009/12/24 15:40:45     kernel:Normal per-cpu:

2009/12/24 15:40:45     kernel:CPU    0: hi:  186, btch:  31 usd:  35

2009/12/24 15:40:45     kernel:CPU    1: hi:  186, btch:  31 usd: 176

2009/12/24 15:40:45     kernel:HighMem per-cpu:

2009/12/24 15:40:45     kernel:CPU    0: hi:  186, btch:  31 usd: 138

2009/12/24 15:40:45     kernel:CPU    1: hi:  186, btch:  31 usd:  33

2009/12/24 15:40:45     kernel:Active:48444 inactive:1888960 dirty:4081 writeback:1073 unstable:0

2009/12/24 15:40:45     kernel:free:32229 slab:70972 mapped:4889 pagetables:523 bounce:0

2009/12/24 15:40:45     kernel:DMA free:2496kB min:80kB low:100kB high:120kB active:0kB inactive:184kB present:16256kB pages_scanned:0 all_unreclaimable? no

2009/12/24 15:40:45     kernel:lowmem_reserve[]: 0 619 8874 8874

2009/12/24 15:40:45     kernel:Normal free:1068kB min:3140kB low:3924kB high:4708kB active:111472kB inactive:90884kB present:633984kB pages_scanned:113 all_unreclaimable? no

2009/12/24 15:40:45     kernel:lowmem_reserve[]: 0 0 66040 66040

2009/12/24 15:40:45     kernel:HighMem free:125352kB min:512kB low:10988kB high:21468kB active:82304kB inactive:7464772kB present:8453120kB pages_scanned:0 all_unreclaimable? no

2009/12/24 15:40:45     kernel:lowmem_reserve[]: 0 0 0 0

2009/12/24 15:40:45     kernel:DMA: 1*4kB 10*8kB 3*16kB 8*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 2500kB

2009/12/24 15:40:45     kernel:Normal: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1020kB

2009/12/24 15:40:45     kernel:HighMem: 2*4kB 4817*8kB 1832*16kB 1484*32kB 1*64kB 3*128kB 9*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 125264kB

2009/12/24 15:40:45     kernel:1922603 total pagecache pages

2009/12/24 15:40:45     kernel:Swap cache: add 11857, delete 11686, find 747/961

2009/12/24 15:40:45     kernel:Free swap  = 12554252kB

2009/12/24 15:40:45     kernel:Total swap = 12582888kB

Our DSS setup:
Mainboard: Srv X7DBE Dual Xeon i5000P 1333MHz DDR2 FB, 6xSATA raid 5, 2xGbE, VGA 16Mb, E-ATX
1 x Xeon E5335 Quad core 1333MHz, 2.0GHz 2x4Mb L2 cache
Controller Areca ARC-1280ML 24 Ch. PCI-E x8 SATA2 RAID6 256Mb DDR2, Areca Battery Backup
24 x Hitachi 1TB SATA2 7200rpm 32Mb cache
8GB ram
1 x QuadPort Intel NIC

3 RAID6 VG with ~50 volumes, ISCSI and NAS, 50-500Gb.

Is there a known kernel bug that could explain the two mail messages?

[symm]

Looks like problem with memory
try running memtest, restart your server and right after the POST, hit 'Esc'
and you'll get options to run the test.**It's best to let the memtest run
until it's completed.

[To-M]

I second that as well to check the memory and run the file system repair util in the Extended tools from Console.