Where Can I learn about iSCSI Settings?

I'm trying to optimize my SAN. I just switched to using Jumbo Frames. My read speed has gone up drastically, but my write speed has gotten a lot worse.

It's very hard because I don't know how to tune it. The default settings on my initiator (Atto XtendSAN) are completely different than the Open-E iSCSI Enterprise's default settings.

What does Data Digest mean? Max Headers? MaxBurstLenght? Immediate Data? Is there a link to a website or a book or something I can read up and understand what this stuff means?

The target disks are formatted HFS+, all the clients are OS X, and I'm using a SAN program called MetaSAN. All this prevents me from using IOmeter, making it even harder.

Or could someone make a recommendation?

The target volumes are RAID 5 w/ 256k block size.

There are 4 client machines, and the target has a four-port 1gb nic. Everything is set to Jumbo Frames 9000 MTU. No bonding or anything like that yet.

The application is video editing. Any suggestions? Thanks.

Have you tested without the Jumbo frames, some switches if not set correctly and the host end not matching can cause some issues. Test without Jumbo frames and with 802.3ad (set the switch as well). Does the RAID controller have the ability to in-crease Cache write values this can help with large file writes. Having a difference in settings of iSCSI target and Initiator can cause issues, set them the same using the default settings from our console located by going to ctrl. + alt. + w select tuning options and iSCSI daemon options (please be careful when experimenting with this as this has already been set properly with our certified initiators). Also check host systems NIC’s that could be causing the slow writes with overhead CPU usage as your reads are fine coming from the server. We certified on our website to use the MAC ATTO iSCSI Mac Initiator--Xtend SAN. Below is the link.

http://www.open-e.com/data_storage_s...ufact=Software

Allot of information is located on the net for the iSCSI settings and there meaning, below is a simple explanation, please review.

http://www.die.net/doc/linux/man/man5/iscsi.conf.5.html

Yes, I've been using it without Jumbo Frames for almost a year. I get about 75MB/s up and down. It's pretty good, but I need to improve it. Also, there are some latency issues. I sometimes get "dropped frames" when capturing video to it. Everyone I've spoken to says the Jumbo Frames is a must for this type of application. My switch and all components are set to use Jumbo Frames.

The disks are nowhere even close to being a bottleneck. It's a 3ware controller, 256kb block size with write cache enabled. And as I said, the write speed only goes down a lot when Jumbo Frames are enabled.

The default settings on Atto XtendSAN v2.0 have all burst lenghts's very high compared to the Open-E's default settings (1048576). The Open-E settings seem to override them anyways.

Thanks for posting that link, but it's really just a description of what the code lines are to configure it. It doesn't explain what things like InitialR2T and ImmediateData mean.

Try to test the system and review the results using this system status for CPU, Memory...
https://x.x.x.x/check_sys/127.0.0.1.html on the iSCSI server (has this been updated to Version 172 Build 2512) . Check to see if all firmwares have been updated. Some testing on all levels will need to be performed. Again, on the internet there are many areas that talk about these subjects from your inquires of what do they mean " InitialR2T and ImmediateData" .

InitialR2T

The InitialR2T key is used to turn off the default use of R2T for
unidirectional and the output part of bidirectional commands, thus
allowing an initiator to start sending data to a target as if it has
received an initial R2T with Buffer Offset=Immediate Data Length and
Desired Data Transfer Length=(min(FirstBurstLength, Expected Data
Transfer Length) - Received Immediate Data Length).

The default action is that R2T is required, unless both the initiator
and the target send this key-pair attribute specifying InitialR2T=No.
Only the first outgoing data burst (immediate data and/or separate
PDUs) can be sent unsolicited (i.e., not requiring an explicit R2T).

ImmediateData

The initiator and target negotiate support for immediate data. To
turn immediate data off, the initiator or target must state its
desire to do so. ImmediateData can be turned on if both the
initiator and target have ImmediateData=Yes.

If ImmediateData is set to Yes and InitialR2T is set to Yes
(default), then only immediate data are accepted in the first burst.

If ImmediateData is set to No and InitialR2T is set to Yes, then the
initiator MUST NOT send unsolicited data and the target MUST reject
unsolicited data with the corresponding response code.

If ImmediateData is set to No and InitialR2T is set to No, then the
initiator MUST NOT send unsolicited immediate data, but MAY send one
unsolicited burst of Data-Out PDUs.

If ImmediateData is set to Yes and InitialR2T is set to No, then the
initiator MAY send unsolicited immediate data and/or one unsolicited
burst of Data-Out PDUs.

MaxRecvDataSegmentLength

The initiator or target declares the maximum data segment length in
bytes it can receive in an iSCSI PDU.

The transmitter (initiator or target) is required to send PDUs with a
data segment that does not exceed MaxRecvDataSegmentLength of the
receiver.

A target receiver is additionally limited by MaxBurstLength for
solicited data and FirstBurstLength for unsolicited data. An
initiator MUST NOT send solicited PDUs exceeding MaxBurstLength nor
unsolicited PDUs exceeding FirstBurstLength (or
FirstBurstLength-Immediate Data Length if immediate data were sent).

MaxBurstLength

The initiator and target negotiate maximum SCSI data payload in bytes
in a Data-In or a solicited Data-Out iSCSI sequence. A sequence
consists of one or more consecutive Data-In or Data-Out PDUs that end
with a Data-In or Data-Out PDU with the F bit set to one.

FirstBurstLength

The initiator and target negotiate the maximum amount in bytes of
unsolicited data an iSCSI initiator may send to the target during the
execution of a single SCSI command. This covers the immediate data
(if any) and the sequence of unsolicited Data-Out PDUs (if any) that
follow the command.

FirstBurstLength MUST NOT exceed MaxBurstLength.

DefaultTime2Wait

The initiator and target negotiate the minimum time, in seconds, to
wait before attempting an explicit/implicit logout or an active task
reassignment after an unexpected connection termination or a
connection reset.

A value of 0 indicates that logout or active task reassignment can be
attempted immediately.

Thanks Tom! That was very helpful.

I mean Todd ;)

Todd,

Thanks for the detailed post, and I also found some other websites with good info. Unfortunately it's a bit over my head. I'm just don't understand the discrepancies between the default settings on Open-E and the initiators' default settings.

Open-E has R2T on by default, and ImmediateData off. Both Atto and MS initiators have the opposite by default.

Open-E default max data segment length (send and receive) is 8k. MS default receive is 512k and Atto is 1M.

Open-E default first burst length is 65k. MS is 512k, Atto is 1M.

Open-E default max burst length is 256k. Again, MS is 512k and Atto is 1M.

Sorry, forgive my ignorance. I'm just confused by the whole thing.

Our engineers decided these where the best tested default settings from many tests. Additionaly alowing you to adjust some of the perameters in our daemon acordingly to be matched with the approved initiatotoers. Many adjustemnst can be made as you are experincing just as long as you keep them the same on both sides. Many other manufactures have different settings as well. I use this site below to perform many different test and this might help in your research for your environment as no two environments are usually the same.

http://www.zvon.org/tmRFC/RFC3720/Output/chapter12.html