iSCSI R3 Enterprise performance tuning

[Raudi]

After i installed the 2.30 i go to the console typed ctrl-alt-x and selected the point 10 - "remove partition".

The 3ware Firmware is not the latest, it is 3.08.00.004 and the current on the web is 3.08.02.005, which has only support for new models and a fix for RAID6. I must connect a floppy to update the firmware. Will try tomorrow.

What the test.log says to the netstat -s and ethtool ethx, i don't see there something wrong:

*-----------------------------------------------------------------------------*
netstat -s
*-----------------------------------------------------------------------------*

Ip:
    939354255 total packets received
    0 forwarded
    0 incoming packets discarded
    939354255 incoming packets delivered
    324964503 requests sent out
Icmp:
    35 ICMP messages received
    0 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 31
        echo requests: 4
    35 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 31
        echo replies: 4
Tcp:
    60171 active connections openings
    60306 passive connection openings
    0 failed connection attempts
    23 connection resets received
    49 connections established
    939343091 segments received
    324956798 segments send out
    221 segments retransmited
    0 bad segments received.
    686 resets sent
Udp:
    10944 packets received
    0 packets to unknown port received.
    0 packet receive errors
    7668 packets sent
TcpExt:
    59746 TCP sockets finished time wait in fast timer
    144181 delayed acks sent
    275 delayed acks further delayed because of locked socket
    Quick ack mode was activated 217840 times
    50422 packets directly queued to recvmsg prequeue.
    11156301 of bytes directly received from prequeue
    864295031 packet headers predicted
    22820 packets header predicted and directly queued to user
    19737188 acknowledgments not containing data received
    22681705 predicted acknowledgments
    2 times recovered from packet loss due to fast retransmit
    10 times recovered from packet loss due to SACK data
    Detected reordering 2 times using time stamp
    2 congestion windows fully recovered
    61 congestion windows partially recovered using Hoe heuristic
    60 congestion windows recovered after partial ack
    0 TCP data loss events
    1 timeouts after SACK recovery
    10 fast retransmits
    1 forward retransmits
    202 other TCP timeouts
    163 DSACKs sent for old packets
    1 connections reset due to unexpected data

*-----------------------------------------------------------------------------*
ethtool eth0 
*-----------------------------------------------------------------------------*

Settings for eth0:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Supports auto-negotiation: Yes
	Advertised link modes:  10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Advertised auto-negotiation: Yes
	Speed: 1000Mb/s
	Duplex: Full
	Port: Twisted Pair
	PHYAD: 1
	Transceiver: internal
	Auto-negotiation: on
	Supports Wake-on: umbg
	Wake-on: d
	Current message level: 0x00000007 (7)
	Link detected: yes

*-----------------------------------------------------------------------------*
ethtool eth1 
*-----------------------------------------------------------------------------*

Settings for eth1:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Supports auto-negotiation: Yes
	Advertised link modes:  10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Advertised auto-negotiation: Yes
	Speed: 1000Mb/s
	Duplex: Full
	Port: Twisted Pair
	PHYAD: 1
	Transceiver: internal
	Auto-negotiation: on
	Supports Wake-on: umbg
	Wake-on: g
	Current message level: 0x00000007 (7)
	Link detected: yes

*-----------------------------------------------------------------------------*
ethtool eth2 
*-----------------------------------------------------------------------------*

Settings for eth2:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Supports auto-negotiation: Yes
	Advertised link modes:  10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Advertised auto-negotiation: Yes
	Speed: 1000Mb/s
	Duplex: Full
	Port: Twisted Pair
	PHYAD: 1
	Transceiver: internal
	Auto-negotiation: on
	Supports Wake-on: umbg
	Wake-on: g
	Current message level: 0x00000007 (7)
	Link detected: yes

*-----------------------------------------------------------------------------*

I wrote a long mail to the support and will call them tomorrow to talk about this Problem. I hope at phone we find a solution...

Regards
Stefan

[To-M]

Correct the netstat -s info is good. What are the iSCSI daemon options for the Target (we did enable WB for the Target - correct - but this would not be much of an improvment - dont worry about that then). Any errors in the logs (error.log, critical_errors or 2 .logs).
If all is good then I would bet the issue to be the RAID controller - check and verify the RAID health or firmware update.

[Raudi]

But how can this be a raid controller issue, when the operating system from the open-e can read from it with 350 MB/s?

Regards
Stefan

[Raudi]

I tested some things:

Used a additional Reaktek GigaBit NIC, but the IOMeter read speed was only 7 MB/s and the write speed like the Intel NIC at 40 MB/s.

I tested with the multiprocessor and sigleprocessor testing kernels, but all the same...

I updated the Firmware of the 3Ware to the last one, but no change.

Some questions:

- What is the better enviroment: Direct links from the target to the initiator with a single cable or is it better to use a switch.

- Can i test the network performance from the initiator to the target and ignore the performance of the raid?

- What must i do to configure the NIC to 1000 FullDuplex fix, no autodetection.

Regards
Stefan

[To-M]

Maybe disabling Flow control will help. Try changing this from the Console Tools in ALT+CTRL+T -> Modify driver option function. Any option will need to restart system. Sometimes forcing the speed value parameters may work for some NICs also look to see what the AutoNeg is set to (Intel NICs are set to 32).

[Raudi]

But what must i enter in the fields for "FlowControl", "AutoNeg", "Duplex" and "Speed" to set them?

I don't think that this will help, because a few hours ago i startet my restore again, this data will go with eth1 into the storage and runs with 70 MBit/s (from check_sys). In ESX i saw a speed of 8 MB/s. Then i started a IOMeter from my workstation which is connected to eth0 and got 30-35 MB/s but in this moment the speed in ESX goes down to 500 KB/s. Wehen i stop IOMeter the speed in ESX goes up again... But i will leave nothing untested.

The best will be when i can made a networking test, to see what performance i got when i made this on one NIC and then on two and final at all three NIC's.

Regards
Stefan

[Raudi]

I configured with now FlowControl and fix at 1000 Full Duplex, but no change...

Can this be a hardware issue?

I use this Supermicro Board: PDSM4

Are there recommendations which board i can use with a 3Ware RAID and some Intel NIC's?

Regards
Stefan