Visit Open-E website
Page 1 of 3 123 LastLast
Results 1 to 10 of 29

Thread: Bad performance with BlockIO

  1. #1
    Join Date
    Jun 2007
    Posts
    84

    Question Bad performance with BlockIO

    Hello,

    at the last weeks i made manny tests, because i had the problem, that the performance was after the last update very bad.

    I used IOmeter with the standard Settings, made a 2000000 blocks (1GB) file and tested with 32KB Seq. 100% read. With one worker i got never more than 35-40 MB/s and when i tryed to use two workers the speed goes down to 10-20 MB/s. Nearly the same i got when i start IOmeter on 2 hosts (at the storage a own NIC) on the first i got 30 MB/s and on the second only 8 MB/s.

    The tool HD Tune shows this very good:



    At 50% i startet this tool on a second machine. This can't be normal that the read performance goes down that way.

    The support was working on this 3 weeks with remote support and the only thing thing they can say is that at other enviroments it is fast. (Send me IOmeter values from a DSS with MPIO, but i use iSCSI R3 Enterprice!) And that they think that this is a Intel network driver issue.

    I was thinking about to use OpenFiler, because in a test OpenFiler 2.2 was much faster on the same hardware. Last night i made manny tests with OpenFiler 2.3, OpenFiler 2.2 and finaly again with the current Open-E with FileIO because i noticed that OpenFiler 2.2 uses FileIO too.

    With a Open-E iSCSI target in FileIO mode i got the performance that i expected. When i start IOmeter i can bring the NIC in the Windows Taskmanager to 99% load. (111 MB/s)

    On 2 hosts at the same time i got 206 MB/s (1st 110 MB/s & 2nd 96 MB/s).

    Where is here the problem? Why is the BlockIO, which should be 30% faster, in my envirement so extremly slow?

    Regards
    Stefan

  2. #2

    Default

    I would have to review the notes from support to find out why. IF they say it is the Intel NIC issue then maybe try to set the parameters for Auto Negotiation.
    On console ALT+CTRL+T->Modify driver options select e1000 option and choose AutoNeg. Set this value to 32. More info about driver options in link below.

    Anyway you can download the Demo-CD for DSS and run a test (not using the USB) then send back the results using the sames ticket number also with MPIO (with Microsoft), I think this is what they meant. This way we can verify it is not just the iSCSI-R3 issue.

    Also was this an upgrade from older version of 2.21 to 2.30, if so try deleting the Unit (VG) and recreate it again. I dont know the case history in which you had with support but what is the type of RAID controller that you are using?

    Keep in mind that there are sometimes isolated incedents where the hw could be the issue and maybe there is a seeting that we need to tweak. Email back to support your findings for OpenFiler as well.

    http://www.intel.com/support/network/sb/cs-009209.htm
    All the best,

    Todd Maxwell


    Follow the red "E"
    Facebook | Twitter | YouTube

  3. #3
    Join Date
    Jun 2007
    Posts
    84

    Default

    Hello Todd!

    All the parameters i tested already and the support told me that some cards have problems. I can't believe it that all my cards have problems. I have a PCIe attached Intel 82573 on board and a Intel Pro 1000XT Server Adapter in one of the PCI-X slots. And i tested a Pro 1000MT Desktop too, but same problem...

    I ask you: When i made my iSCSI targets with FileIO and with this i got a 99% load of the network card in windows (111 MB/s) when using the MS iSCSI Initiator but with BlockIO i got never more than 35-40 MB/s. Can this be a network card configuration issue?

    A other thing i wrote the suooprt and this points against a network issue too:

    While using BlockIO i downloaded a log during a restore from LTO tape with 500 MB/Min. In the log hdparm reported this:
    Timing buffered disk reads: 282 MB in 3.04 seconds = 92.70 MB/sec

    Without a runnung tape restore i got:
    Timing buffered disk reads: 1098 MB in 3.00 seconds = 365.94 MB/sec

    Now my data is again restoring from tape and i downloaded a log again. The restore runs faster than ever i had with BlockIO, now the restore runs with 850 MB/Min:
    Timing buffered disk reads: 910 MB in 3.00 seconds = 303.29 MB/sec

    And the upgrade from the 2.21 to the 2.30 was the problem. I made it, deleted all data on the drive from the console and made all new from the Web-Gui. After that the system was slow. (On the same hardware i installed the OpenFiler after that i must made the volumes new, again.)

    I'm sending all my informations to the support. But i think they didn't take me seriously.

    One week ago they want a remote session, i should made some tests with iometer. I got no feedback what they find out. Next time i called they asked me about my controller and wanted a log. They didn't know that i send already a log? Everytime i call i think i begin for new...

    Is here someone who uses the iSCSI R3 Enterprise with BlockIO and this works?

    Can i use FileIO and BlockIO mixed? I have created 2 targets with FileIO on my storage and 900GB of free space. For a test, can i switch the target type to BlockIO and create a 3rd target hat uses BlockIO? If not then i'm finished with testing. Currently i made the 3rd restore of my 1,7 TB of data...

    Greetings
    Stefan

  4. #4
    Join Date
    Jun 2007
    Posts
    84

    Default

    Something i noticed... Can it possible that in BlockIO the local memory will not used for caching? If i remember right, a frew weeks ago i was wondering about much free memory on my system. Now all memory is used for system cache.

    Stefan

  5. #5

    Default

    Did you test with the Demo-CD - DSS yet? Also send your results to support with new logs.
    In blockIO we use only cache specified for devices - fileIO use filesystem cache and devices cache. The default maximum of device cache in 32bit kernel is limited actually to about 1GB.

    This was emailed to me today asking them to perform some test to verify there speed with iSCSI-R3 with 4x1Gb - MPIO from Microsoft. There must be something that needs to be tweaked on the hardware (firmware, NIC driver....), they testing with the Demo-CD then send the logs using the same ticket number.

    hardware specification

    Target

    iSCSI-R3 Ent. Ver. 2.30 build 2820
    CPU 2 x QUAD (Xeon) 2Ghz
    Ram: 2Gb
    Controller: LSI Logic SAS
    Disc: 8x36GB (Raid0)
    NIC: Intel Corporation Pro/1000 PT Quad Port LP server Adapter
    Information: SSR212mc2

    Client

    Windows 2003 server with service pack 1
    CPU: Xeon 1.86 Ghz
    Ram: 2Gb
    NIC: Intel Corporation Pro/1000 Pt Quat Port LP server Adapter
    initiator: Windows iSCSI Initiator 2.05 build 2392


    Performance Result

    IOmeter MPIO , out_io=100 row-device
    1worker 64k seq write io=3963,72 mb=247.73 cpu=70.78 1worker 64k seq read
    io=5951,91 mb=371,99 cpu=81,76

    2workers 64k seq write io=3776,17 mb=236.01 cpu=76,43 2workers 64k seq read
    io=5368,14 mb=335,51 cpu=75,64

    1worker 512 seq write io=700,37 mb=0.34 cpu=2,35 1worker 512 seq read
    io=9264,68 mb=4,52 cpu=21,42

    2workers 512 seq write io=1478.56 mb=0.72 cpu=5,76 2workers 512 seq read
    io=13180,47 mb=6,44 cpu=22,88
    All the best,

    Todd Maxwell


    Follow the red "E"
    Facebook | Twitter | YouTube

  6. #6
    Join Date
    Jun 2007
    Posts
    84

    Default

    First my question: Is it possible to create a BlockIO target in a unit whre are already existing 2 FileIO targets? I don't want delete my data again only for a test.

    And i tryed to download the Demo-CD but the link doesn't work. I logged in with my serial, selected the Demo-CD and then i got a error Message that the page cannot be displayed...

    Greetings
    Stefan

  7. #7

    Default

    You cannot select a specified target to be BlockIO. As stated in release notes it is recommended to back up your data and remove volumes and recreate them.

    Thanks for the link issue - you are correct there is a problem we will correct it.

    Email me todd.maxwell@open-e.com and I will provide another site as this is not to be published on the Forum.
    All the best,

    Todd Maxwell


    Follow the red "E"
    Facebook | Twitter | YouTube

  8. #8
    Join Date
    Jun 2007
    Posts
    84

    Default

    The release notes told nothing about mixed enviroments, they told only about volumes and not about volume groups or units.

    At this moment my default iSCSI target type is FileIO, i have free space in my volume group. If i understand right i can now switch to BlockIO on the console and create a new volume in the free space of the volume group. The result will be that i have my 2 old iSCSI volumes in FileIO and a new one in BlockIO.

    Correct?

    I got a demo CD, i used my e-mail address to get a access code, with this code i can download the CD.

    Greetings
    Stefan

  9. #9

    Default

    New volumes are created in Block IO and as stated below in order to use old volumes that are created in File IO the data will need to backed up then restored on new volumes. You can still use your existing volumes as they where created in File IO but you will not be able to mix for Replication. In the end you would be better to keep them either File IO or Block IO.

    Release Notes:

    NEW:
    * IMPORTANT : In ver 1.30 and 1.32 New iSCSI, default volume creation is done in block-IO in contrast to the older version that was file-IO.
    In order to create new iSCSI volume in "old" file-IO mode, please switch the default in console tools menu: ctrl-alt-w --> Tuning options-->iSCSI daemon options-->iSCSI volume type.
    Block-IO mode is about 30% faster then File-IO and the target volume size is exactly equal to defined iSCSI volume size (in file-io the target size is a bit smaller then defined iSCSI volume size).
    Additionally, initialization is no-longer required as with the old file-io volumes.
    In order to migrate your data from the file-IO to block-IO volume, the data must be backed up from the existing iSCSI volumes and then restored into newly created iSCSI volumes.
    NOTE: please verify data integrity prior to deleting old iSCSI volumes.
    Volume replication is only possible between to similar volumes (e.g. block-io <-> block-io or file-io <-> file-io). Volume replication between old file-io and block-io volumes are NOT possible.
    All the best,

    Todd Maxwell


    Follow the red "E"
    Facebook | Twitter | YouTube

  10. #10
    Join Date
    Jun 2007
    Posts
    84

    Default

    Ysterday i got the 2.32 and the small update for the intel drivers from the support to test it.

    The problem is still the same. I still think it is a cache problem.

    What i'm thinking:

    With BlockIO there will be used a small cache so the read and write access is not optimized. With sequential read or write with one client or IOmeter with one worker there is no problem. But when i use more workers, i access the RAID with 2 clients at the same time or i made random access the performance goes extemely down.

    With FileIO there will be used a large cache size (3,8 GB) so the writes can better ordered before writing to disk. So there will be more time for reading.

    I made a test with IOmeter, this profile i used: smalltest.zip

    Open-E S-ATA-Storage with FileIO:
    64KB Random Read/Write Mix: 120 MB/s (80 MB read/40 MB write)
    4KB Random Read/Write Mix: 15130 IO/s
    1MB Sequential Read: 106 MB/s
    1MB Sequential Write: 111 MB/s

    Open-E S-ATA-Storage with BlockIO:
    64KB Random Read/Write Mix: 14 MB/s (9,5 MB read/4,5 MB write)
    4KB Random Read/Write Mix: 350 IO/s
    1MB Sequential Read: 93 MB/s
    1MB Sequential Write: 110 MB/s

    My single local S-ATA disk is in random IO faster then the iSCSI storage in BlockIO...

    Greetings
    Stefan

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •