Visit Open-E website
Page 1 of 3 123 LastLast
Results 1 to 10 of 25

Thread: iSCSI R3 Enterprise performance tuning

  1. #1
    Join Date
    Jun 2007
    Posts
    84

    Unhappy iSCSI R3 Enterprise performance tuning

    Hello,

    i still have performance problems. Yesterday i updated to the current version and deleted the complete unit to made new iSCSI volumes. After this i copyed the vmdk files from my local VMFS back to the VMFS on the iSCSI Volume. A 6 GB file takes 11:15 minutes, this will be 9 MB/Sec.

    If i remember right, the copy of the file in the other direction to a local single S-ATA drive was much faster. At this time the restore from tape is running so i think i got no useable value when i test it now. The backup to the LTO1 tape runs with 900MB/Min and now the restore only with 550MB/Min. (2 days remaining to restore all the data.)

    At the performance chart on the esx host for the network card i used for the ESX iSCSI initaor i got never more than 30 MBps. The current value during restore is 10 MBps.

    When i use IOMeter in a VM on a RAW mapped 2 TB iSCSI Volume i got for read and write a value of 35 MB/Sec.

    What can i do to optimise the performance?

    My enviroment:

    Storage
    Supermicro PDSM4 Mainboard with a Pentium D 930 (3 GHz Dual Core) and 4 GB memory.
    3Ware 9550SXU 16 Ch. S-ATA II (with BBU in performance mode) with 6x Seagate 750 GB HDD's

    2x ESX Host
    Supermicro X7DBE Mainboard with two Xeon E5320 (1,86 GHz Quad Core) and 6 GB memory.

    For iSCSI communication i use direct cables from the two on board NIC's of the storage to each ESX host.

    Best regards
    Stefan

  2. #2
    Join Date
    Jun 2007
    Posts
    84

    Default

    Note:
    The Virtual Infrastructure Client seem to show the wrong value. There is a "0" missing.

    The current Data Transmit Rate is shown as 12000 KBps, this will be 1,5 MB/s and can't be correct. This must be 120000 KBps corresponding to the value i read on the check_sys page on the Open-E.

    Regards
    Stefan

  3. #3
    Join Date
    Jun 2007
    Posts
    84

    Default

    Argh... The KBps are "KB per Second" and not "KBit per Second"!

    But this doesn't change anything at my problem....

  4. #4

    Default

    Can you test without the VM iscsi connection and just use direct with a MS initiator (MS XP or 2003) to see if it is the iSCSI-R3. Also check the tests.log in the log file from Status > Hardware > Download Logs then look for the hdparm -t /dev/sdx to see what speeds we are getting from the RAID set.
    All the best,

    Todd Maxwell


    Follow the red "E"
    Facebook | Twitter | YouTube

  5. #5
    Join Date
    Jun 2007
    Posts
    84

    Default

    Now i canceled my restore to made some tests:

    To copy a 8 GB file from a single 500 GB drive (connected to a MegaRaid SATA 150-6) to iSCSI it take 13 min and 23 sec. The other direction only 5 min. and 43 sec.

    The other informations:

    I made a new volume and mounted it from my Windows XP with the MS Initiator. With IOMeter i got the same values i got in the VM. 30-35 while reading and 30-40 during write... (32K 100% seq. read or write)

    From the tests.log:

    --- Physical volume ---
    PV Name /dev/sdb
    VG Name vg+3ware
    PV Size 3.41 TB / not usable 0
    Allocatable yes
    PE Size (KByte) 32768
    Total PE 111757
    Free PE 12
    Allocated PE 111745

    hdparm -t /dev/sdb

    /dev/sdb:
    Timing buffered disk reads: 1036 MB in 3.00 seconds = 345.31 MB/sec

    This looks good. Is there a way to test only the network connection? (like with iperf)

    Regards
    Stefan

  6. #6
    Join Date
    Jun 2007
    Posts
    84

    Default

    It must be a local problem on my iSCSI box. Now i restarted the restore and during this i tested again with IOMeter from my Workstation. The value for read is now between 0 and 5 MB/s...

    The workstation is connected to a Intel GigaBit Server Adapter which is placed in a PCI-X slot. The ESX Hosts are connected to the both on board Intel GigaBit NIC's.

    Can i made some other tests? Where is the bottleneck?

    Regards
    Stefan

  7. #7

    Default

    Check the tests.log and see what the netstat -s and ethtool ethx has. Also is the 3Ware firmware up to date. Is the iSCSI-R3 version 2.30 and was this upgraded from older version? If it was updated from older version did you delete the Logical Volumes and VG' then recreate (from release notes).

    Try to test with different Kernel. Go to Console screen - then Console Tools CTRL.
    + ALT. + T select Boot Options - Select System Architecture - Multi
    processor server system (TESTING).
    All the best,

    Todd Maxwell


    Follow the red "E"
    Facebook | Twitter | YouTube

  8. #8
    Join Date
    Jun 2007
    Posts
    84

    Default

    After i installed the 2.30 i go to the console typed ctrl-alt-x and selected the point 10 - "remove partition".

    The 3ware Firmware is not the latest, it is 3.08.00.004 and the current on the web is 3.08.02.005, which has only support for new models and a fix for RAID6. I must connect a floppy to update the firmware. Will try tomorrow.

    What the test.log says to the netstat -s and ethtool ethx, i don't see there something wrong:

    Code:
    *-----------------------------------------------------------------------------*
    netstat -s
    *-----------------------------------------------------------------------------*
    
    Ip:
        939354255 total packets received
        0 forwarded
        0 incoming packets discarded
        939354255 incoming packets delivered
        324964503 requests sent out
    Icmp:
        35 ICMP messages received
        0 input ICMP message failed.
        ICMP input histogram:
            destination unreachable: 31
            echo requests: 4
        35 ICMP messages sent
        0 ICMP messages failed
        ICMP output histogram:
            destination unreachable: 31
            echo replies: 4
    Tcp:
        60171 active connections openings
        60306 passive connection openings
        0 failed connection attempts
        23 connection resets received
        49 connections established
        939343091 segments received
        324956798 segments send out
        221 segments retransmited
        0 bad segments received.
        686 resets sent
    Udp:
        10944 packets received
        0 packets to unknown port received.
        0 packet receive errors
        7668 packets sent
    TcpExt:
        59746 TCP sockets finished time wait in fast timer
        144181 delayed acks sent
        275 delayed acks further delayed because of locked socket
        Quick ack mode was activated 217840 times
        50422 packets directly queued to recvmsg prequeue.
        11156301 of bytes directly received from prequeue
        864295031 packet headers predicted
        22820 packets header predicted and directly queued to user
        19737188 acknowledgments not containing data received
        22681705 predicted acknowledgments
        2 times recovered from packet loss due to fast retransmit
        10 times recovered from packet loss due to SACK data
        Detected reordering 2 times using time stamp
        2 congestion windows fully recovered
        61 congestion windows partially recovered using Hoe heuristic
        60 congestion windows recovered after partial ack
        0 TCP data loss events
        1 timeouts after SACK recovery
        10 fast retransmits
        1 forward retransmits
        202 other TCP timeouts
        163 DSACKs sent for old packets
        1 connections reset due to unexpected data
    
    *-----------------------------------------------------------------------------*
    ethtool eth0 
    *-----------------------------------------------------------------------------*
    
    Settings for eth0:
    	Supported ports: [ TP ]
    	Supported link modes:   10baseT/Half 10baseT/Full 
    	                        100baseT/Half 100baseT/Full 
    	                        1000baseT/Full 
    	Supports auto-negotiation: Yes
    	Advertised link modes:  10baseT/Half 10baseT/Full 
    	                        100baseT/Half 100baseT/Full 
    	                        1000baseT/Full 
    	Advertised auto-negotiation: Yes
    	Speed: 1000Mb/s
    	Duplex: Full
    	Port: Twisted Pair
    	PHYAD: 1
    	Transceiver: internal
    	Auto-negotiation: on
    	Supports Wake-on: umbg
    	Wake-on: d
    	Current message level: 0x00000007 (7)
    	Link detected: yes
    
    *-----------------------------------------------------------------------------*
    ethtool eth1 
    *-----------------------------------------------------------------------------*
    
    Settings for eth1:
    	Supported ports: [ TP ]
    	Supported link modes:   10baseT/Half 10baseT/Full 
    	                        100baseT/Half 100baseT/Full 
    	                        1000baseT/Full 
    	Supports auto-negotiation: Yes
    	Advertised link modes:  10baseT/Half 10baseT/Full 
    	                        100baseT/Half 100baseT/Full 
    	                        1000baseT/Full 
    	Advertised auto-negotiation: Yes
    	Speed: 1000Mb/s
    	Duplex: Full
    	Port: Twisted Pair
    	PHYAD: 1
    	Transceiver: internal
    	Auto-negotiation: on
    	Supports Wake-on: umbg
    	Wake-on: g
    	Current message level: 0x00000007 (7)
    	Link detected: yes
    
    *-----------------------------------------------------------------------------*
    ethtool eth2 
    *-----------------------------------------------------------------------------*
    
    Settings for eth2:
    	Supported ports: [ TP ]
    	Supported link modes:   10baseT/Half 10baseT/Full 
    	                        100baseT/Half 100baseT/Full 
    	                        1000baseT/Full 
    	Supports auto-negotiation: Yes
    	Advertised link modes:  10baseT/Half 10baseT/Full 
    	                        100baseT/Half 100baseT/Full 
    	                        1000baseT/Full 
    	Advertised auto-negotiation: Yes
    	Speed: 1000Mb/s
    	Duplex: Full
    	Port: Twisted Pair
    	PHYAD: 1
    	Transceiver: internal
    	Auto-negotiation: on
    	Supports Wake-on: umbg
    	Wake-on: g
    	Current message level: 0x00000007 (7)
    	Link detected: yes
    
    *-----------------------------------------------------------------------------*
    I wrote a long mail to the support and will call them tomorrow to talk about this Problem. I hope at phone we find a solution...

    Regards
    Stefan

  9. #9

    Default

    Correct the netstat -s info is good. What are the iSCSI daemon options for the Target (we did enable WB for the Target - correct - but this would not be much of an improvment - dont worry about that then). Any errors in the logs (error.log, critical_errors or 2 .logs).
    If all is good then I would bet the issue to be the RAID controller - check and verify the RAID health or firmware update.
    All the best,

    Todd Maxwell


    Follow the red "E"
    Facebook | Twitter | YouTube

  10. #10
    Join Date
    Jun 2007
    Posts
    84

    Default

    But how can this be a raid controller issue, when the operating system from the open-e can read from it with 350 MB/s?

    Regards
    Stefan

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •