DSS FC is very fast.

I just installed a DSS server at a customer's site. It had 20 GB of cache (16GB of system cache and 4GB on the 1680ix Areca card), dual quad-core (eight cores), and 16 1TB SAS drives. It also had a (dual-port) 4Gb fibre channel card in it (for using in target mode).

I ran an SQLIO test on a test system which had a small test volume mounted on it from the DSS. When using 4KB block sizes and random reads (with 8 outstanding requests), it got 14000 I/O! That's about 55 MB/s random I/O. Not bad at all. Not only that, but that's only running with 1Gb fibre channel (that's all the test system had). Granted, it's all cached, but if you're running a small database about 10GB or less, it could very well be all cached anyway, or at least, the part you're actually using could be all cached.

You're supposed to benchmark by using a test file at least 2-4 times as big as your cache, but I only used a 1GB test file, since I didn't have time to test it again. Still, that's pretty good!

Anyways, I'm very happy with this outcome. Also, the average read latency, according to SQLIO, is 0 milliseconds. That's right. Zero. Obviously, SQLIO needs to add an option to display the results in microseconds (or nanoseconds), but looks pretty good.

Linear reads are around 96 MB/s, so that's obviously bumping up against the (initiator-side) test system's 1Gb FC interface (which is only using a 33MHz/32-bit PCI bus).

I'm sorry I didn't get the chance to test the system with a test-file that is bigger than the cache and with a 4Gb FC pci-e card on both ends. I also should've got write speeds, too. It would've been very interesting. Next time, I guess!

VERY COOL!!!

I am going to pass this onto the team! Please keep us up to date on the rests of the test.

Well, unfortunately I didn't run any more tests. I wish I had had a whole day to run tests, but I didn't.

I'm thinking, though, that if I lowered the block size to 512B and used a full 4Gb between two machines, over 100,000 random I/Os is not out of the question if you can fit everything in the cache. We have some other options for motherboards in our standard box, too, which have 16 slots, which would let us cheaply add 64GB of ECC RAM. There are also relatively inexpensive boxes out there (1U's, even), which have 32 slots, allowing a cheap 128GB of (ECC) RAM. We could market these as disk-backed memory appliances or database-acceleration boxes if there was a way to keep everything in cache (well, once the database starts up, everything important would get put in cache, so that's not really important). If we had two of them with autofailover and cache-coherent DRBD, you have effectively a highly-available memory appliance for much less expensive than less-reliable (because no autofailover) proprietary memory appliances.

Except for the fibre-channel autofailover, we could basically do this right now. Granted, it'd be much lower latency if DRBD worked with Infiniband (RDMA, not the IPoIB). But, for customers who don't care too much about high-availability or keeping track of every single write operation, this would work fine.

So I see you are using the Seagate 1TB SAS drives. How are those working out for you?

Great. I guess if I were doing a strict cost/benefit analysis, the SATA 1TB drives would come out a little bit on top (since they're so cheap), but when a little extra performance matters but you can't afford the ridiculous prices for 10k/15k drives, they work very well. Besides, we have lots of experience working with Seagate for well over a decade.

Overall, the 1TB SAS drives work very well and are much cheaper comparatively than any other SAS drive. They also have 32MB of cache, which is more than other SAS drives.

Do they suffer the same firmware problems that the ES2 SATA drives have had the last year or so? I'm not loving Seagate this year...

Well, our distributor had already patched in the newer (fixed) firmware, so we didn't have that problem at all. I had been prepared to flash it myself, but it was already done. Really, we haven't suffered any drive failures from the firmware problem. Stuff like this happens every once in a while to all the manufacturers, and it isn't as severe as, say, problems that we've had with the 400GB Hitachi "Deathstar" drives. We still like Seagate a lot.

Alright, so I've been doing some testing on a smaller system using FC. Unfortunately, the initiator side still has only 1Gb FC with 32bit/33Mhz PCI-bus, but at least that gives me a way to compare performance to the DSS FC system we've already installed on our customer's site.

Basically, random read I/O/s are about 14,000 from the cache. The DSS target has only 1GB of system memory, so I used a smaller test file of only 200MB. This DSS target is using a software RAID 5 with 3 drives, so there's not much performance coming from the drives themselves

Target side:
DSS b3278 32-bit
Dual-core pentium 4 (Netburst) 3GHz (no Hyperthreading). 1GB of system RAM. 3x500GB 7200 rpm Barracuda SATA drives (I think they have 32MB of cache, otherwise it's 16MB each). Dual-port 4Gb qlogic fibrechannel card with Pci-E x4. 10GB FC volume with 4KB blocksize, default options (i.e. write-back enabled, not write-through).

Initiator side:
Windows 2003 Server.
Pentium 4 2GHz (Netburst crap). 1GB system RAM. some really old single-port 1Gb Qlogic Fibrechannel card plugged into a 32bit/33Mhz PCI slot.

sqlio -kR -s60 -frandom -o16 -b4 -LS d:\testfile.dat 1 0x0 200

or something like that. I have to run it for a lot longer beforehand in order to make sure that the whole test file is in the DSS's system cache.

Anyways, I consistently get about 13980 or so, which is within 5% of the much higher spec'd system I used before. The difference there was that it had 4 times as many cores (and they were using the Core-flavored Xeons which probably had a much higher FSB and DDR2 vs DDR).

So, it seems that ~14000 random read I/Os is to be expected when using 1Gb FC with 4kB blocks and a test file that fits entirely in the system cache. When I use a larger test file which doesn't fit in the cache, the random read I/Os on this system drop to around 150 to 300.

So, if I made a system with, say, 128GB of system memory in 1U, then I could get a usable high-performance SAN without having to fill a rack up with hundreds of 15K SAS drives and then short-stroking them. The only bad part so far is that I'd have to wait until the cache was filled until I could expect high performance.

PS, we have another system coming in that's identical to the DSS target system (i.e. dual-core pentium4, etc.), so I should be able to test how well this works with volume replication.

Stay tuned for write performance and performance using iSCSI!

I just ran sqlio while using an iscsi volume (only using a single 1Gb ethernet connection, jumbo frames not enabled). It was pretty slow for random read I/O in cache. In fact, it was about only 470 I/O when I ran it this way (with a 200MB test file):

sqlio -kR -s60 -frandom -o128 -b4 -LS -Fparam.txt

The param.txt file has only this in it: "H:\testfile.dat 1 0x0 200", which is where the test file is located (i.e. we are testing the D volume right now, which happens to be iSCSI) and how many threads to run (one) and the bitmask that sets processor affinity (does not apply, since there's only one core) and how big of a test file in megabytes. When I did both the FC volume and the iSCSI volume, I used NTFS with 4096Byte blocksize. The initiator is MS S/W iSCSI 2.08. The iSCSI target is block I/O mounted with writeback cache enabled. The FC volume uses 4096 byte blocksize.

Whereas when I use a FC volume over a 1Gb FC interface, it's about 14000 random read I/O.

I've done this a few times now, and it seems quite obvious that FC is far superior for random I/O reads from cache. Granted, I am only able to test one thread at a time right now, since my initiator side is only single-core. Oh well.

I tried running the same thing on the FC volume right after this test (but first running a sequential sqlio run to put the benchmark file in the DSS system cache, as in the iSCS volume test). I get over 14,000 random read I/Os!

BTW, the easiest way to make sure the whole benchmark test file is in the cache while using sqlio is to just run sqlio in sequential mode beforehand once or twice for a long enough for it to read the entire test file. If this were not a benchmark but a production environment with a cache large enough for, say, an entire mysql database to fit in, one could just "dd if=/dev/sda1 of=/dev/null bs=4k" to make sure everything is in cache before you start using it. This could be done on the DSS side or the client side. The DSS side would be a little faster, but it doesn't really matter that much, since you'd only rarely have to do it.


PS, I'll try to see what difference jumbo frames makes, if I can enable it.

Here are some of my results with writing to a fibre channel volume (1Gb link):
http://www.massmountain.com/images/s...ndomwrites.png

I get over 10000 random write iops. Not bad. But, as the size of the test file goes up, the random write iops drop dramatically.

These are the results with an iSCSI volume (1Gb eth with no jumbo frames):
http://www.massmountain.com/images/s...ndomwrites.png

The random write iops never got higher than 117. This is without jumbo frames. My on-board ethernet card doesn't support jumbo frames, but I have a card that does, so I'll use that next. Even so, this is rediculously low performance. There's probably something I did wrong. I'll try to find it. 117 IOPS sounds like it isn't using any of the DSS's system RAM for caching and is just going straight to the disks.

Also, BTW, that's with using a Block I/O iSCSI volume. I'll try it next with a File I/O volume.

THIS IS OUTSTANDING WORK!!! Just to let you know that the engineers are all looking into your work :)

They wanted to know if you can test with the Atlanta version.

I think I will propose to post your results on our website and see if we can create a section for best performance tests with our customers and partners using FC, iSCSI and NAS.

Thanks Mass Storage - you guy's ROCK!

I finished a huge battery of tests using different test file sizes for comparison. For very small test file sizes (10MB and 40MB) with 4k random writes, fibre channel outperforms iscsi up to 80 times! :eek:

The left side is fibre channel, the right is iscsi.
http://www.massmountain.com/images/s...efcvsiscsi.png

The left side is fibre channel, the right is iscsi.

I have more results coming tomorrow.

Test setting the iSCSI daemon settings for the Target to the following below. When getting better performance - see if we can replace this map with a current one.

Please go to the console and enter crtl + alt + w then select Tuning options then iSCSI
daemon options then Target options then select Target of choice.

maxRecvDataSegmentLen=262144
MaxBurstLength=16776192
Maxxmitdatasegment=262144
maxoutstandingr2t=8
InitialR2T=No
ImmediateData=Yes

Many are reading this thread so we should keep it shorter, but really good job on the comparison!!

Robotbeat!!

Check this posting on speeds reported with Open-E DSS Benchmark with PIC!
Would be good to check with him. I will let him know of your resutls as well.

Server
DELL 860 with 6/i
RAID0 WD GP 750G X2
Open - DSS Version: 5.0up60.7101.3511 32bit
FC HBA:Qlogic QLA2344 2Gbps PCI-X in target mode


http://forum.open-e.com/showthread.php?t=1319

Quote:

---

Originally Posted by Robotbeat

I finished a huge battery of tests using different test file sizes for comparison. For very small test file sizes (10MB and 40MB) with 4k random writes, fibre channel outperforms iscsi up to 80 times! :eek:

---

Hi Robotbeat,

there must be something wrong with your setup. look at my results:

http://funkmasters.de/dss/dss.JPG

Our DSS is a dual XEON 3,2 GHz, 8 GB RAM, ARC 1280 - 2GB, using block-IO and 1GB iSCSI connections.
We run 4 1TB SATA drives in Raid 10 configuration, in total 5 volumes with 10 TB.
This is no test setup, so we have some load (21 ESX Servers, 136 VMs, 44 SAP Test Servers) on this DSS - the values may be not exactly reproducable.

As you can see, our old and busy setup beats your FC DSS easily in higher file sizes.
It seems to be more constant compared to your very high values with low file sizes and low values at high file sizes.

We actually have no issues with bad performance using iSCSI and block-IO. Its not fast as hell but it is very acceptable with high load.

Best Regards,
Lutz

Thanks, lufu. It's great to get a comparison point!

Yeah, there probably is something wrong with my iSCSI set up.

Part of the problem is that I am using a software RAID 5 using only 3 drives (2+parity).

Not using jumbo frames, either.

Both sides are also using pentium 4 chips with netburst and only one thread of sqlio (since the initiator side has only one cpu core). Shouldn't be THAT slow, though. I tried tuning the iSCSI target daemon settings, but I haven't got much better results.

Part of my whole point was to see how well FC works when just working from cache, since you could load up a 1U server with 100GB of cache instead of investing in lots of SAS drives.

BTW, I really like the Areca controllers. That's what we've used in our setup.

Here are the 4K random write IO results from a single drive hooked up to a 3ware controller with 512MB of raid cache and 2GB of system cache exported via 4Gb FC:
http://www.massmountain.com/images/s...ndwritesoe.png

It makes sense that the smaller test files are so high performance and the larger ones are so much slower, since we're limited to a single sata drive on the backend, so we'd be lucky to get more than 60MB/s sustained.