Bad performance with BlockIO

[Raudi]

Hello,

at the last weeks i made manny tests, because i had the problem, that the performance was after the last update very bad.

I used IOmeter with the standard Settings, made a 2000000 blocks (1GB) file and tested with 32KB Seq. 100% read. With one worker i got never more than 35-40 MB/s and when i tryed to use two workers the speed goes down to 10-20 MB/s. Nearly the same i got when i start IOmeter on 2 hosts (at the storage a own NIC) on the first i got 30 MB/s and on the second only 8 MB/s.

The tool HD Tune shows this very good:



At 50% i startet this tool on a second machine. This can't be normal that the read performance goes down that way.

The support was working on this 3 weeks with remote support and the only thing thing they can say is that at other enviroments it is fast. (Send me IOmeter values from a DSS with MPIO, but i use iSCSI R3 Enterprice!) And that they think that this is a Intel network driver issue.

I was thinking about to use OpenFiler, because in a test OpenFiler 2.2 was much faster on the same hardware. Last night i made manny tests with OpenFiler 2.3, OpenFiler 2.2 and finaly again with the current Open-E with FileIO because i noticed that OpenFiler 2.2 uses FileIO too.

With a Open-E iSCSI target in FileIO mode i got the performance that i expected. When i start IOmeter i can bring the NIC in the Windows Taskmanager to 99% load. (111 MB/s)

On 2 hosts at the same time i got 206 MB/s (1st 110 MB/s & 2nd 96 MB/s).

Where is here the problem? Why is the BlockIO, which should be 30% faster, in my envirement so extremly slow?

Regards
Stefan

[To-M]

I would have to review the notes from support to find out why. IF they say it is the Intel NIC issue then maybe try to set the parameters for Auto Negotiation.
On console ALT+CTRL+T->Modify driver options select e1000 option and choose AutoNeg. Set this value to 32. More info about driver options in link below.

Anyway you can download the Demo-CD for DSS and run a test (not using the USB) then send back the results using the sames ticket number also with MPIO (with Microsoft), I think this is what they meant. This way we can verify it is not just the iSCSI-R3 issue.

Also was this an upgrade from older version of 2.21 to 2.30, if so try deleting the Unit (VG) and recreate it again. I dont know the case history in which you had with support but what is the type of RAID controller that you are using?

Keep in mind that there are sometimes isolated incedents where the hw could be the issue and maybe there is a seeting that we need to tweak. Email back to support your findings for OpenFiler as well.

http://www.intel.com/support/network/sb/cs-009209.htm

[Raudi]

Hello Todd!

All the parameters i tested already and the support told me that some cards have problems. I can't believe it that all my cards have problems. I have a PCIe attached Intel 82573 on board and a Intel Pro 1000XT Server Adapter in one of the PCI-X slots. And i tested a Pro 1000MT Desktop too, but same problem...

I ask you: When i made my iSCSI targets with FileIO and with this i got a 99% load of the network card in windows (111 MB/s) when using the MS iSCSI Initiator but with BlockIO i got never more than 35-40 MB/s. Can this be a network card configuration issue?

A other thing i wrote the suooprt and this points against a network issue too:

While using BlockIO i downloaded a log during a restore from LTO tape with 500 MB/Min. In the log hdparm reported this:
Timing buffered disk reads: 282 MB in 3.04 seconds = 92.70 MB/sec

Without a runnung tape restore i got:
Timing buffered disk reads: 1098 MB in 3.00 seconds = 365.94 MB/sec

Now my data is again restoring from tape and i downloaded a log again. The restore runs faster than ever i had with BlockIO, now the restore runs with 850 MB/Min:
Timing buffered disk reads: 910 MB in 3.00 seconds = 303.29 MB/sec

And the upgrade from the 2.21 to the 2.30 was the problem. I made it, deleted all data on the drive from the console and made all new from the Web-Gui. After that the system was slow. (On the same hardware i installed the OpenFiler after that i must made the volumes new, again.)

I'm sending all my informations to the support. But i think they didn't take me seriously.

One week ago they want a remote session, i should made some tests with iometer. I got no feedback what they find out. Next time i called they asked me about my controller and wanted a log. They didn't know that i send already a log? Everytime i call i think i begin for new...

Is here someone who uses the iSCSI R3 Enterprise with BlockIO and this works?

Can i use FileIO and BlockIO mixed? I have created 2 targets with FileIO on my storage and 900GB of free space. For a test, can i switch the target type to BlockIO and create a 3rd target hat uses BlockIO? If not then i'm finished with testing. Currently i made the 3rd restore of my 1,7 TB of data...

Greetings
Stefan

[Raudi]

Something i noticed... Can it possible that in BlockIO the local memory will not used for caching? If i remember right, a frew weeks ago i was wondering about much free memory on my system. Now all memory is used for system cache.

Stefan

[To-M]

Did you test with the Demo-CD - DSS yet? Also send your results to support with new logs.
In blockIO we use only cache specified for devices - fileIO use filesystem cache and devices cache. The default maximum of device cache in 32bit kernel is limited actually to about 1GB.

This was emailed to me today asking them to perform some test to verify there speed with iSCSI-R3 with 4x1Gb - MPIO from Microsoft. There must be something that needs to be tweaked on the hardware (firmware, NIC driver....), they testing with the Demo-CD then send the logs using the same ticket number.

hardware specification

Target

iSCSI-R3 Ent. Ver. 2.30 build 2820
CPU 2 x QUAD (Xeon) 2Ghz
Ram: 2Gb
Controller: LSI Logic SAS
Disc: 8x36GB (Raid0)
NIC: Intel Corporation Pro/1000 PT Quad Port LP server Adapter
Information: SSR212mc2

Client

Windows 2003 server with service pack 1
CPU: Xeon 1.86 Ghz
Ram: 2Gb
NIC: Intel Corporation Pro/1000 Pt Quat Port LP server Adapter
initiator: Windows iSCSI Initiator 2.05 build 2392


Performance Result

IOmeter MPIO , out_io=100 row-device
1worker 64k seq write io=3963,72 mb=247.73 cpu=70.78 1worker 64k seq read
io=5951,91 mb=371,99 cpu=81,76

2workers 64k seq write io=3776,17 mb=236.01 cpu=76,43 2workers 64k seq read
io=5368,14 mb=335,51 cpu=75,64

1worker 512 seq write io=700,37 mb=0.34 cpu=2,35 1worker 512 seq read
io=9264,68 mb=4,52 cpu=21,42

2workers 512 seq write io=1478.56 mb=0.72 cpu=5,76 2workers 512 seq read
io=13180,47 mb=6,44 cpu=22,88

[Raudi]

First my question: Is it possible to create a BlockIO target in a unit whre are already existing 2 FileIO targets? I don't want delete my data again only for a test.

And i tryed to download the Demo-CD but the link doesn't work. I logged in with my serial, selected the Demo-CD and then i got a error Message that the page cannot be displayed...

Greetings
Stefan