Bad performance with BlockIO

Hello,

at the last weeks i made manny tests, because i had the problem, that the performance was after the last update very bad.

I used IOmeter with the standard Settings, made a 2000000 blocks (1GB) file and tested with 32KB Seq. 100% read. With one worker i got never more than 35-40 MB/s and when i tryed to use two workers the speed goes down to 10-20 MB/s. Nearly the same i got when i start IOmeter on 2 hosts (at the storage a own NIC) on the first i got 30 MB/s and on the second only 8 MB/s.

The tool HD Tune shows this very good:

http://www.raudonis.de/files/iscsi4.jpg

At 50% i startet this tool on a second machine. This can't be normal that the read performance goes down that way.

The support was working on this 3 weeks with remote support and the only thing thing they can say is that at other enviroments it is fast. (Send me IOmeter values from a DSS with MPIO, but i use iSCSI R3 Enterprice!) And that they think that this is a Intel network driver issue.

I was thinking about to use OpenFiler, because in a test OpenFiler 2.2 was much faster on the same hardware. Last night i made manny tests with OpenFiler 2.3, OpenFiler 2.2 and finaly again with the current Open-E with FileIO because i noticed that OpenFiler 2.2 uses FileIO too.

With a Open-E iSCSI target in FileIO mode i got the performance that i expected. When i start IOmeter i can bring the NIC in the Windows Taskmanager to 99% load. (111 MB/s)

On 2 hosts at the same time i got 206 MB/s (1st 110 MB/s & 2nd 96 MB/s).

Where is here the problem? Why is the BlockIO, which should be 30% faster, in my envirement so extremly slow?

Regards
Stefan

I would have to review the notes from support to find out why. IF they say it is the Intel NIC issue then maybe try to set the parameters for Auto Negotiation.
On console ALT+CTRL+T->Modify driver options select e1000 option and choose AutoNeg. Set this value to 32. More info about driver options in link below.

Anyway you can download the Demo-CD for DSS and run a test (not using the USB) then send back the results using the sames ticket number also with MPIO (with Microsoft), I think this is what they meant. This way we can verify it is not just the iSCSI-R3 issue.

Also was this an upgrade from older version of 2.21 to 2.30, if so try deleting the Unit (VG) and recreate it again. I dont know the case history in which you had with support but what is the type of RAID controller that you are using?

Keep in mind that there are sometimes isolated incedents where the hw could be the issue and maybe there is a seeting that we need to tweak. Email back to support your findings for OpenFiler as well.

http://www.intel.com/support/network/sb/cs-009209.htm

Hello Todd!

All the parameters i tested already and the support told me that some cards have problems. I can't believe it that all my cards have problems. I have a PCIe attached Intel 82573 on board and a Intel Pro 1000XT Server Adapter in one of the PCI-X slots. And i tested a Pro 1000MT Desktop too, but same problem...

I ask you: When i made my iSCSI targets with FileIO and with this i got a 99% load of the network card in windows (111 MB/s) when using the MS iSCSI Initiator but with BlockIO i got never more than 35-40 MB/s. Can this be a network card configuration issue?

A other thing i wrote the suooprt and this points against a network issue too:

While using BlockIO i downloaded a log during a restore from LTO tape with 500 MB/Min. In the log hdparm reported this:
Timing buffered disk reads: 282 MB in 3.04 seconds = 92.70 MB/sec

Without a runnung tape restore i got:
Timing buffered disk reads: 1098 MB in 3.00 seconds = 365.94 MB/sec

Now my data is again restoring from tape and i downloaded a log again. The restore runs faster than ever i had with BlockIO, now the restore runs with 850 MB/Min:
Timing buffered disk reads: 910 MB in 3.00 seconds = 303.29 MB/sec

And the upgrade from the 2.21 to the 2.30 was the problem. I made it, deleted all data on the drive from the console and made all new from the Web-Gui. After that the system was slow. (On the same hardware i installed the OpenFiler after that i must made the volumes new, again.)

I'm sending all my informations to the support. But i think they didn't take me seriously.

One week ago they want a remote session, i should made some tests with iometer. I got no feedback what they find out. Next time i called they asked me about my controller and wanted a log. They didn't know that i send already a log? Everytime i call i think i begin for new...

Is here someone who uses the iSCSI R3 Enterprise with BlockIO and this works?

Can i use FileIO and BlockIO mixed? I have created 2 targets with FileIO on my storage and 900GB of free space. For a test, can i switch the target type to BlockIO and create a 3rd target hat uses BlockIO? If not then i'm finished with testing. Currently i made the 3rd restore of my 1,7 TB of data...

Greetings
Stefan

Something i noticed... Can it possible that in BlockIO the local memory will not used for caching? If i remember right, a frew weeks ago i was wondering about much free memory on my system. Now all memory is used for system cache.

Stefan

Did you test with the Demo-CD - DSS yet? Also send your results to support with new logs.
In blockIO we use only cache specified for devices - fileIO use filesystem cache and devices cache. The default maximum of device cache in 32bit kernel is limited actually to about 1GB.

This was emailed to me today asking them to perform some test to verify there speed with iSCSI-R3 with 4x1Gb - MPIO from Microsoft. There must be something that needs to be tweaked on the hardware (firmware, NIC driver....), they testing with the Demo-CD then send the logs using the same ticket number.

hardware specification

Target

iSCSI-R3 Ent. Ver. 2.30 build 2820
CPU 2 x QUAD (Xeon) 2Ghz
Ram: 2Gb
Controller: LSI Logic SAS
Disc: 8x36GB (Raid0)
NIC: Intel Corporation Pro/1000 PT Quad Port LP server Adapter
Information: SSR212mc2

Client

Windows 2003 server with service pack 1
CPU: Xeon 1.86 Ghz
Ram: 2Gb
NIC: Intel Corporation Pro/1000 Pt Quat Port LP server Adapter
initiator: Windows iSCSI Initiator 2.05 build 2392


Performance Result

IOmeter MPIO , out_io=100 row-device
1worker 64k seq write io=3963,72 mb=247.73 cpu=70.78 1worker 64k seq read
io=5951,91 mb=371,99 cpu=81,76

2workers 64k seq write io=3776,17 mb=236.01 cpu=76,43 2workers 64k seq read
io=5368,14 mb=335,51 cpu=75,64

1worker 512 seq write io=700,37 mb=0.34 cpu=2,35 1worker 512 seq read
io=9264,68 mb=4,52 cpu=21,42

2workers 512 seq write io=1478.56 mb=0.72 cpu=5,76 2workers 512 seq read
io=13180,47 mb=6,44 cpu=22,88

First my question: Is it possible to create a BlockIO target in a unit whre are already existing 2 FileIO targets? I don't want delete my data again only for a test.

And i tryed to download the Demo-CD but the link doesn't work. I logged in with my serial, selected the Demo-CD and then i got a error Message that the page cannot be displayed...

Greetings
Stefan

You cannot select a specified target to be BlockIO. As stated in release notes it is recommended to back up your data and remove volumes and recreate them.

Thanks for the link issue - you are correct there is a problem we will correct it.

Email me todd.maxwell@open-e.com and I will provide another site as this is not to be published on the Forum.

The release notes told nothing about mixed enviroments, they told only about volumes and not about volume groups or units.

At this moment my default iSCSI target type is FileIO, i have free space in my volume group. If i understand right i can now switch to BlockIO on the console and create a new volume in the free space of the volume group. The result will be that i have my 2 old iSCSI volumes in FileIO and a new one in BlockIO.

Correct?

I got a demo CD, i used my e-mail address to get a access code, with this code i can download the CD.

Greetings
Stefan

New volumes are created in Block IO and as stated below in order to use old volumes that are created in File IO the data will need to backed up then restored on new volumes. You can still use your existing volumes as they where created in File IO but you will not be able to mix for Replication. In the end you would be better to keep them either File IO or Block IO.

Release Notes:

NEW:
* IMPORTANT : In ver 1.30 and 1.32 New iSCSI, default volume creation is done in block-IO in contrast to the older version that was file-IO.
In order to create new iSCSI volume in "old" file-IO mode, please switch the default in console tools menu: ctrl-alt-w --> Tuning options-->iSCSI daemon options-->iSCSI volume type.
Block-IO mode is about 30% faster then File-IO and the target volume size is exactly equal to defined iSCSI volume size (in file-io the target size is a bit smaller then defined iSCSI volume size).
Additionally, initialization is no-longer required as with the old file-io volumes.
In order to migrate your data from the file-IO to block-IO volume, the data must be backed up from the existing iSCSI volumes and then restored into newly created iSCSI volumes.
NOTE: please verify data integrity prior to deleting old iSCSI volumes.
Volume replication is only possible between to similar volumes (e.g. block-io <-> block-io or file-io <-> file-io). Volume replication between old file-io and block-io volumes are NOT possible.

Ysterday i got the 2.32 and the small update for the intel drivers from the support to test it.

The problem is still the same. I still think it is a cache problem.

What i'm thinking:

With BlockIO there will be used a small cache so the read and write access is not optimized. With sequential read or write with one client or IOmeter with one worker there is no problem. But when i use more workers, i access the RAID with 2 clients at the same time or i made random access the performance goes extemely down.

With FileIO there will be used a large cache size (3,8 GB) so the writes can better ordered before writing to disk. So there will be more time for reading.

I made a test with IOmeter, this profile i used: smalltest.zip

Open-E S-ATA-Storage with FileIO:
64KB Random Read/Write Mix: 120 MB/s (80 MB read/40 MB write)
4KB Random Read/Write Mix: 15130 IO/s
1MB Sequential Read: 106 MB/s
1MB Sequential Write: 111 MB/s

Open-E S-ATA-Storage with BlockIO:
64KB Random Read/Write Mix: 14 MB/s (9,5 MB read/4,5 MB write)
4KB Random Read/Write Mix: 350 IO/s
1MB Sequential Read: 93 MB/s
1MB Sequential Write: 110 MB/s

My single local S-ATA disk is in random IO faster then the iSCSI storage in BlockIO...

Greetings
Stefan

Hello,

one question:

Today i have the time to test the DSS. What happens with my data? Can i use the DSS and then the iSCSI Enterprise again without loosing data?

Greetings
Stefan

If using the DSS with iSCSI-R3 Enterprise then yes.

O.K.

I used the DSS Demo CD and the performance is the same. Sequential read with one worker is fast and with two workers the value goes down after a few seconds.

Best regards
Stefan

I have asked engineers to review this issue - they may need logs from the DSS using your ticket # from last communications.

FYI: Engineers have confirmed that fileIO can by much faster in random access.

OK, so in my Enviroment there is in the current version FileIO the better choice.

Will there be changes in the future that more cache will be used in BlockIO? (When i'm right that the cachig in FileIO results the more performance in random IO.)

Best regards

Stefan

Our Research department will review this but they will need to time to schedule testing.

Hi Todd,

did you hear something?

Regards
Stefan

This is going to take time for them to work on this as there are many projects in front that they have to complete first. Nothing I can do now.

Hi,

today I did update our iSCSI R3 Enterprise from 2.21 to 5.0 and did create new RAID and a volume with Block I/O. We use the iSCSI storage for our VMWare ESX 3.5 server virtual maschines. Inside a virtual maschine I use IOMeter to measure the disk performance and it was very weak. I was playing with the tuning options (InitialR2t = no, ImmediateData = yes) without performance boost. Now I deleted the Block I/O volume und made a new File I/O volume and get acceptable performance after playing with the tuning options.

With Block I/O I get 22 MB/s read and 5 MB/s write and with File I/O I get 57 MB/s read and 48 MB/s write (IOMeter is set to 100% sequential, 64k, 100% read or 100%write).

Any ideas?

Thanks,
Marcus

One more question: I was trying this free DSS Lite last month, checked the iSCSI part and got good performance (IOMeter inside virtual maschine). Does DSS Lite use Block I/O by default and if yes, why it was so fast and iSCSI R3 Enterprise is so slow?

DSS LITE use Block IO by default....And as far as I know both the systems are the same...

you were using same settings???

Hard to say. I was using DSS Lite on another maschine, same board and NICs but older processor and less RAM. I did not configure much and I'm not sure if I created a RAID. I just had this DSS Lite to find out best settings inside the tuning options. Can RAID be the problem? With the iSCSI R3 Enterprise I use software RAID 5 on 4 harddisks, left-symmetric, 64k chunk size.

So used a H/W RAID on one and a software for the other??

If hardware is the same and software is the same.. what is the difference here?

I was told from support to use File IO, as it seems to be using the cache better for VMware.

Didn't you get a message box during creating the iSCSI volume, that Open-E suggest you to use file IO when this vlume is using with vmware?

BlockIO is only fast when a single task is working on the disks. When multiple tasks are working (like a few VM's) block IO is not useable...

Best regards
Stefan

Thanks for your feedbacks. I was not using hardware RAID on DSS Lite or iSCSI R3. On iSCSI R3 I use software RAID 5 and on DSS Lite I'm not sure what I did configure. But I know I used default Block IO with DSS Lite and got good performance while testing with one VM and IOMeter. Testing few virtual disks with IOMeter I never did. Now I cannot play with my DSS Lite because the activation is gone and I dont want to start the activation procedure with putting the maschine in front of our firewall...
When I use Block IO with iSCSI R3, I got no message box telling me that File IO is suggested with VMWare.

I will use File IO with my iSCSI R3 and give up playing with Block IO.

What I wish is more support from Open-E giving us information what are the best settings using their products with VMWare. I was reading Block IO is 30% faster than File IO. So I moved all our VMs to another storage, made the update, recreated a volume with Block IO just to realize the performance is pretty poor and File IO is recommended for use with VMWare.

Best regards
Marcus

open-e fixed that issue.. Now in the new version they write a script to show a pop-up box stating/recommnded to use file io if u r using VMWare...

I just moves all my vm's (esx 3.5) to fileio from blockio and i got much better performance.

thanks for all in this post.

Radui Wrote:
> BlockIO is only fast when a single task is working on the disks.

We found this to be incorrect. We use iSCSI target as a Virtual Tape Libray (VTL). The data in the VTL is also backed up to tape for offsite storage. This is single access, long sequential read, when being offloaded to physical tape and the peformance of file I/O is double that of block I/O. Block I/O gives no more than 30MB/sec, where as File I/O we get 60MB/sec.

We have done extensive testing with IO Meter and file I/O wins everytime. I don't understand why block I/O is the default when it's clearly slower (at least in our case anyway).