snippet from dmesg due to restriction of 10000 characters:
Using IPI Shortcut mode
Freeing unused kernel memory: 240k freed
squashfs: version 3.2-r2 (2007/01/15) Phillip Lougher
aufs 20070903
attempt to access beyond end of device
loop0: rw=0, want=66, limit=8
isofs_fill_super: bread failed, dev=loop0, iso_blknum=16, block=32
attempt to access beyond end of device
loop0: rw=0, want=68, limit=8
attempt to access beyond end of device
loop0: rw=0, want=1252, limit=8
attempt to access beyond end of device
loop0: rw=0, want=1028, limit=8
UDF-fs: No partition found (1)
XFS: bad magic number
XFS: SB validate failed
Vendor: TTI-MSA Model: USB 2.0 MD Rev: PMAP
Type: Direct-Access ANSI SCSI revision: 00
SCSI device sda: 985088 512-byte hdwr sectors (504 MB)
sda: Write Protect is off
sda: Mode Sense: 23 00 00 00
SCSI device sda: 985088 512-byte hdwr sectors (504 MB)
sda: Write Protect is off
sda: Mode Sense: 23 00 00 00
sda: sda1
sd 0:0:0:0: Attached scsi removable disk sda
usb-storage: device scan complete
UDF-fs: No VRS found
XFS: bad magic number
XFS: SB validate failed
UDF-fs: No VRS found
XFS: bad magic number
XFS: SB validate failed
kjournald starting. Commit interval 5 seconds
EXT3 FS on dm-0, internal journal
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
attempt to access beyond end of device
dm-1: rw=0, want=66, limit=8
isofs_fill_super: bread failed, dev=dm-1, iso_blknum=16, block=32
attempt to access beyond end of device
dm-1: rw=0, want=68, limit=8
attempt to access beyond end of device
dm-1: rw=0, want=1252, limit=8
attempt to access beyond end of device
dm-1: rw=0, want=1028, limit=8
UDF-fs: No partition found (1)
XFS: bad magic number
XFS: SB validate failed
UDF-fs: No VRS found
XFS: bad magic number
XFS: SB validate failed
UDF-fs: No VRS found
XFS: bad magic number
XFS: SB validate failed
UDF-fs: No VRS found
XFS: bad magic number
XFS: SB validate failed
UDF-fs: No VRS found
XFS: bad magic number
XFS: SB validate failed
attempt to access beyond end of device
dm-6: rw=0, want=354, limit=352
isofs_fill_super: bread failed, dev=dm-6, iso_blknum=88, block=176
UDF-fs: No VRS found
XFS: bad magic number
XFS: SB validate failed
I've created 3 volumes which will be accessed by two servers via RHCS/clvm. I've formatted all 3 with GFS2. Currently are both servers connected but only one is accessing one target.
After researching this with the XFS: SB validate failed & XFS: bad magic number errors this could be issues with your Volume. Not sure if you have any data residing on these volumes but if you can backup the data and reconfigure the Unit (RAID set) and start over.
I would recommend using function from Extended Console Tools - "Clear contents of units" in order to delete VG and LV configuration (reboot will happen). Then in the WebGUI add the unit again to the storage.
I've recreated the unit as you suggested, plugged the remaining NIC into a seperate switch wher only DSS and both servers are connected to. The proble is still persistent :-|
ATM I'm a little bit pissed since recreating the setup, clvm volumes and stuff took nearly three hours.
Oh, when this timeout happens the Xen-DomU, which IS on the iSCSI-volume, stops responding...
dmesg:
SCSI device sda: 985088 512-byte hdwr sectors (504 MB)
sda: Write Protect is off
sda: Mode Sense: 23 00 00 00
sda: sda1
sd 0:0:0:0: Attached scsi removable disk sda
usb-storage: device scan complete
UDF-fs: No VRS found
XFS: bad magic number
XFS: SB validate failed
UDF-fs: No VRS found
XFS: bad magic number
XFS: SB validate failed
kjournald starting. Commit interval 5 seconds
EXT3 FS on dm-0, internal journal
ext3_orphan_cleanup: deleting unreferenced inode 2526
EXT3-fs: dm-0: 1 orphan inode deleted
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
attempt to access beyond end of device
dm-1: rw=0, want=66, limit=8
isofs_fill_super: bread failed, dev=dm-1, iso_blknum=16, block=32
attempt to access beyond end of device
dm-1: rw=0, want=68, limit=8
attempt to access beyond end of device
dm-1: rw=0, want=1252, limit=8
attempt to access beyond end of device
dm-1: rw=0, want=1028, limit=8
UDF-fs: No partition found (1)
XFS: bad magic number
XFS: SB validate failed
UDF-fs: No VRS found
XFS: bad magic number
XFS: SB validate failed
UDF-fs: No VRS found
XFS: bad magic number
XFS: SB validate failed
UDF-fs: No VRS found
XFS: bad magic number
XFS: SB validate failed
UDF-fs: No VRS found
XFS: bad magic number
XFS: SB validate failed
attempt to access beyond end of device
dm-6: rw=0, want=354, limit=352
isofs_fill_super: bread failed, dev=dm-6, iso_blknum=88, block=176
UDF-fs: No VRS found
XFS: bad magic number
XFS: SB validate failed
As I know this [cmnd_abort (1143)] is related to the writing process. If disks are high loaded sometimes could the iSCSI reach timeout before writing goes.
So the iSCSI initiator should retry the writing.
Same command will be send when a disk hardware problem slowdown the writing - not only in high load.
The server is an Opteron 185 or 2218 (dualcore), 4GB Ram and 3ware 9550 4port HBA.
Disks are four WD3200 configured to raid5 on the 3ware. OK, I know that the 3ware suck on Linux, but we're using them on our other identical servers too.
The raid is fine for me, no problems reported by the HBA.
One thing I've noticed is load and CPU-Load when writing to the array: CPU goes up to 50% when writing with one initiator, load goes up to 3 or 4.