DSS V 5.0up62.7101.3518

[citscits]

Hi there,

Just had the storage box freeze completely.

Have pull off the error log - How easy is this to read.

Can anyone start me off by pointing me in the right direction.

Thanks in advance

Paul

2011/06/10 12:57:33|<MRMON251> Controller ID: 0 VD is now DEGRADED VD0
2011/06/10 12:57:33|<MRMON251> Controller ID: 0 VD is now DEGRADED VD0
2011/06/27 20:55:48|Pid: 17243, comm: istiod2 Not tainted 2.6.27.10-oe64-00000-g7c8c1b2 #4
2011/06/27 20:55:48|
2011/06/27 20:55:48|Call Trace:
2011/06/27 20:55:48|[<ffffffff80273048>] oom_kill_process+0x118/0x210
2011/06/27 20:55:48|[<ffffffff80272ce3>] badness+0x163/0x1e0
2011/06/27 20:55:48|[<ffffffff80273556>] out_of_memory+0x1b6/0x230
2011/06/27 20:55:48|[<ffffffff802751dd>] __alloc_pages_internal+0x3cd/0x430
2011/06/27 20:55:48|[<ffffffff80292ebd>] cache_alloc_refill+0x2bd/0x580
2011/06/27 20:55:48|[<ffffffff80292bde>] kmem_cache_alloc+0x7e/0xa0
2011/06/27 20:55:48|[<ffffffffa01d640e>] cmnd_alloc+0x2e/0x100 [iscsi_trgt]
2011/06/27 20:55:48|[<ffffffffa01d64fe>] iscsi_cmnd_create_rsp_cmnd+0x1e/0x60 [iscsi_trgt]
2011/06/27 20:55:48|[<ffffffffa01d6a51>] send_data_rsp+0xc1/0x1d0 [iscsi_trgt]
2011/06/27 20:55:48|[<ffffffffa01df30e>] disk_execute_cmnd+0xbe/0x130 [iscsi_trgt]
2011/06/27 20:55:48|[<ffffffffa01daf46>] worker_thread+0x156/0x1f0 [iscsi_trgt]
2011/06/27 20:55:48|[<ffffffff8022feb0>] default_wake_function+0x0/0x10
2011/06/27 20:55:48|[<ffffffff8022ff06>] __wake_up_common+0x46/0x80
2011/06/27 20:55:48|[<ffffffff8022feb0>] default_wake_function+0x0/0x10
2011/06/27 20:55:48|[<ffffffffa01dadf0>] worker_thread+0x0/0x1f0 [iscsi_trgt]
2011/06/27 20:55:48|[<ffffffff80249d3c>] kthread+0x6c/0xa0
2011/06/27 20:55:48|[<ffffffff8020d1c9>] child_rip+0xa/0x11
2011/06/27 20:55:48|[<ffffffff80249cd0>] kthread+0x0/0xa0
2011/06/27 20:55:48|[<ffffffff8020d1bf>] child_rip+0x0/0x11
2011/06/27 20:55:48|
2011/06/27 20:55:48|DMA per-cpu:
2011/06/27 20:55:48|CPU 0: hi: 0, btch: 1 usd: 0
2011/06/27 20:55:48|CPU 1: hi: 0, btch: 1 usd: 0
2011/06/27 20:55:48|CPU 2: hi: 0, btch: 1 usd: 0
2011/06/27 20:55:48|CPU 3: hi: 0, btch: 1 usd: 0
2011/06/27 20:55:48|DMA32 per-cpu:
2011/06/27 20:55:48|CPU 0: hi: 186, btch: 31 usd: 74
2011/06/27 20:55:48|CPU 1: hi: 186, btch: 31 usd: 162
2011/06/27 20:55:48|CPU 2: hi: 186, btch: 31 usd: 165
2011/06/27 20:55:48|CPU 3: hi: 186, btch: 31 usd: 181
2011/06/27 20:55:48|Normal per-cpu:
2011/06/27 20:55:48|CPU 0: hi: 186, btch: 31 usd: 153
2011/06/27 20:55:48|CPU 1: hi: 186, btch: 31 usd: 113
2011/06/27 20:55:48|CPU 2: hi: 186, btch: 31 usd: 149
2011/06/27 20:55:48|CPU 3: hi: 186, btch: 31 usd: 155
2011/06/27 20:55:48|Active:431470 inactive:541805 dirty:0 writeback:0 unstable:0
2011/06/27 20:55:48|free:4474 slab:13002 mapped:6683 pagetables:750 bounce:0
2011/06/27 20:55:48|DMA free:8228kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:7124kB pages_scanned:0 all_unreclaimable? yes
2011/06/27 20:55:48|lowmem_reserve[]: 0 3503 4007 4007
2011/06/27 20:55:48|DMA32 free:8820kB min:7076kB low:8844kB high:10612kB active:1546224kB inactive:1906356kB present:3587584kB pages_scanned:5963075 all_unreclaimable? yes
2011/06/27 20:55:48|lowmem_reserve[]: 0 0 504 504
2011/06/27 20:55:48|Normal free:848kB min:1016kB low:1268kB high:1524kB active:179656kB inactive:260864kB present:516096kB pages_scanned:673197 all_unreclaimable? yes
2011/06/27 20:55:48|lowmem_reserve[]: 0 0 0 0
2011/06/27 20:55:48|DMA: 3*4kB 3*8kB 4*16kB 4*32kB 3*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 8228kB
2011/06/27 20:55:48|DMA32: 203*4kB 1*8kB 0*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8820kB
2011/06/27 20:55:48|Normal: 0*4kB 2*8kB 2*16kB 1*32kB 0*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 848kB
2011/06/27 20:55:48|957321 total pagecache pages
2011/06/27 20:55:48|0 pages in swap cache
2011/06/27 20:55:48|Swap cache stats: add 0, delete 0, find 0/0
2011/06/27 20:55:48|Free swap = 0kB
2011/06/27 20:55:48|Total swap = 0kB
2011/06/27 22:00:22|Pid: 17238, comm: istd2 Not tainted 2.6.27.10-oe64-00000-g7c8c1b2 #4
2011/06/27 22:00:22|
2011/06/27 22:00:22|Call Trace:
2011/06/27 22:00:22|[<ffffffff80273048>] oom_kill_process+0x118/0x210
2011/06/27 22:00:22|[<ffffffff80272ce3>] badness+0x163/0x1e0
2011/06/27 22:00:22|[<ffffffff80273556>] out_of_memory+0x1b6/0x230
2011/06/27 22:00:22|[<ffffffff802751dd>] __alloc_pages_internal+0x3cd/0x430
2011/06/27 22:00:22|[<ffffffffa01d6095>] tio_alloc+0x95/0x100 [iscsi_trgt]
2011/06/27 22:00:22|[<ffffffffa01d90fa>] cmnd_rx_start+0x93a/0x9c0 [iscsi_trgt]
2011/06/27 22:00:22|[<ffffffff8061f287>] tcp_ioctl+0xe7/0x180
2011/06/27 22:00:22|[<ffffffffa01d9e8f>] istd+0x49f/0x11a0 [iscsi_trgt]
2011/06/27 22:00:22|[<ffffffff8022b2e7>] set_next_entity+0x47/0x50
2011/06/27 22:00:22|[<ffffffff8061f460>] tcp_sendpage+0x0/0x6a0
2011/06/27 22:00:22|[<ffffffff8022ff06>] __wake_up_common+0x46/0x80
2011/06/27 22:00:22|[<ffffffffa01d99f0>] istd+0x0/0x11a0 [iscsi_trgt]
2011/06/27 22:00:22|[<ffffffff80249d3c>] kthread+0x6c/0xa0
2011/06/27 22:00:22|[<ffffffff8020d1c9>] child_rip+0xa/0x11
2011/06/27 22:00:22|[<ffffffff80249cd0>] kthread+0x0/0xa0
2011/06/27 22:00:22|[<ffffffff8020d1bf>] child_rip+0x0/0x11
2011/06/27 22:00:22|
2011/06/27 22:00:22|DMA per-cpu:
2011/06/27 22:00:22|CPU 0: hi: 0, btch: 1 usd: 0
2011/06/27 22:00:22|CPU 1: hi: 0, btch: 1 usd: 0
2011/06/27 22:00:22|CPU 2: hi: 0, btch: 1 usd: 0
2011/06/27 22:00:22|CPU 3: hi: 0, btch: 1 usd: 0
2011/06/27 22:00:22|DMA32 per-cpu:
2011/06/27 22:00:22|CPU 0: hi: 186, btch: 31 usd: 139
2011/06/27 22:00:22|CPU 1: hi: 186, btch: 31 usd: 157
2011/06/27 22:00:22|CPU 2: hi: 186, btch: 31 usd: 157
2011/06/27 22:00:22|CPU 3: hi: 186, btch: 31 usd: 172
2011/06/27 22:00:22|Normal per-cpu:
2011/06/27 22:00:22|CPU 0: hi: 186, btch: 31 usd: 169
2011/06/27 22:00:22|CPU 1: hi: 186, btch: 31 usd: 156
2011/06/27 22:00:22|CPU 2: hi: 186, btch: 31 usd: 185
2011/06/27 22:00:22|CPU 3: hi: 186, btch: 31 usd: 165
2011/06/27 22:00:22|Active:117587 inactive:855548 dirty:0 writeback:0 unstable:0
2011/06/27 22:00:22|free:4665 slab:13022 mapped:6462 pagetables:733 bounce:0
2011/06/27 22:00:22|DMA free:8228kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:7124kB pages_scanned:0 all_unreclaimable? yes
2011/06/27 22:00:22|lowmem_reserve[]: 0 3503 4007 4007
2011/06/27 22:00:22|DMA32 free:9412kB min:7076kB low:8844kB high:10612kB active:239720kB inactive:3214180kB present:3587584kB pages_scanned:3573003 all_unreclaimable? no
2011/06/27 22:00:22|lowmem_reserve[]: 0 0 504 504
2011/06/27 22:00:22|Normal free:1020kB min:1016kB low:1268kB high:1524kB active:230628kB inactive:208012kB present:516096kB pages_scanned:1029231 all_unreclaimable? yes
2011/06/27 22:00:22|lowmem_reserve[]: 0 0 0 0
2011/06/27 22:00:22|DMA: 3*4kB 3*8kB 4*16kB 4*32kB 3*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 8228kB
2011/06/27 22:00:22|DMA32: 377*4kB 14*8kB 3*16kB 0*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 9412kB
2011/06/27 22:00:22|Normal: 39*4kB 2*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1020kB
2011/06/27 22:00:22|957227 total pagecache pages
2011/06/27 22:00:22|0 pages in swap cache
2011/06/27 22:00:22|Swap cache stats: add 0, delete 0, find 0/0
2011/06/27 22:00:22|Free swap = 0kB
2011/06/27 22:00:22|Total swap = 0kB

[Al-S]

You should past this at DSS section not in DSSv6 as DSSv5 = named as DSS.

Your system faced a low space of memory, try to add more memory to the system, or do some tuning options.

Also it seems that you do not have swap memory, are you using an external storage?
if yes that will not allow you to have a swap chache memory, and it is recommended that you add more RAM to your computer.

Also your system had a call trace issue, that means you have to check your RAID Controller health and your NIC cards and other network devices such as cables, switch, router, etc.., also to update your RAID Controller firmware to the latest update. And your drivers in general should be updated ( update to the latest build of DSS gives you the latest update of drivers that available at the date of the latest build release ).

[StamfordRob]

i had my NAS freeze up 2 nights ago and today this is the same errors we are seeing on our events..

Our NAS is backend for vCenter clusters. running out 8GB 802.3ad bond.

history..
* 1 wk ago we added 8GB to the Digiliant NAS
* upgraded to 'small update 4xxx'
* enabled Jumbo Frames on switch and OPENe.
* added tuning-
*** ipfrag high from 262144 to 524288
*** ipfrag low from 196608 to 393216
*** Jumbo Frames to 9000
*** NFS Daemons from 64 to 96

2 nts ago the server froze so we rolled back to 3518 thinking we found a memory leak.. we also upgraded the RIAD controller 52445 to latest firmware 18252..

today i looked at events and it seems to be shaking itself to death with these errors..

thanks in advance..

rob

[Gr-R]

*Thread moved to proper forum*

If your going to stay on DSSV5, you should be running the latest build: 4562

The original post has a degraded RAID, and while its rebuilding, this can affect many things. A light load on the system is preferred during this period.

Due to various firmware changes on the RAID, and drivers in DSSV5, you may have reached an incompatible point here.

Can you try the booting the trial version of DSSV6, to see if your problems are resolved?
Tuning is important as well as the bond type being used. V6 can allow a wider range of tweaks in this regard.

[ghalehame]

Is there any updated or recent version which release recent you know what I mean !

[ghalehame]

I think there is no new updates that's why nobody reply us.

[Gr-R]

I think there is no new updates that's why nobody reply us.

There are no more updates on Version 5, it is end-of-life product.

[Al-S]

Steps that should help:

- Check RAID controller health.
- Update its firmware to latest version.
- Add more memory.
- Change the Controller if possible.
- Update to latest DSSv5 build available ( no more new updates released ).
- Try the DSSv6 Trial version, and see if you face the same issues.