Printable View
A couple of questions here:
1. what are good reasons to do this?
2. Why does it take so darn long?
3. Why did my memory usage skyrocket once the process started
I'm trying to init a RAID10 volume on an areca 1680xi-24 SAS controller. The volume 2500 GB and is made up of 15 drives and it's the system is very slow right now, its been running about an hour and is only 8% complete......
UPDATE:
So at this point my system is almost frozen and it's been running overnight (the volume init)
So I'm really upset about this because this means that if I init a volume I bring the system to it's knees.
Initilization is made for data protection. Basically when you initilize your volume you fill it with zeros, then when you write over the volume the space that has no data will be protected.
Its a low level format, and many big companies are using this, EMC for instance. Its a good feature. It may take time for big volumes, but its so important..
Maybe you need to add memory here.. or break down the volume
Okay. So if I had more memory I would not have run into this issue? Because the box is unusable at this point and I didn't expect this...
This means that adding and init'ng additional volumes would be out of the question on a production system because it would die on the floor
mmm.. This should not be true.. I create volume 250G and they took some time but then they work fine.
What version you are using?
I'm using Version: 5.0.DB46000000.3084
Perhaps I need some extra ram and another CPU. What I notice is that when I init a volume, all my available memory, 1GB+ gets cached and I start swapping a small bit
I have 2GB in this server - which i thought would have been sufficient. I'll need to test with a bit more ram.
-Errol
UPDATE:
So I init'ed a smaller volume. It was 750GB. Afterwards, I looked at the stats in Open-E. For the duration of my init, the load average skyrocketed to 6 and my CPU was maxed out.
Is this how it's supposed to work? I'd rather us block io and suffer the performance woes because that is horrible.
Any takers on this question?
Just my 2cents worth...
The initialization process writes/initializes the single large file that becomes the iSCSI Lun under File-IO mode. It does seem to write at "full speed", so bottle-necks may be exposed (disk, mem, cpu etc.). Your 2GB of memory I would have thought is a little small for a 2.5GB File-IO Lun. It will certainy take many hours, and in your case, have an effect on other traffic, tho I dont recall our systems being affecting traffic quite that much. Initialization of the Lun file is a good idea.
Just another random thought re mem & vol sizes. If you ever need to run a repair on a large volume (via the console), you MUST have plenty of memory, otherwise the xfs repair utility used can never complete and leave you stranded. In fact, really large volumes require 64-bit operation and plenty of Ram - something Open-E doesn't document. This may only affect NAS volumes, but something to remember.
I dont wish to be show my age, but a "low level format" is not something we do these days to drives, more back in the heady days of 20MB half- or full-height MFM/RLL drives! Anyone remember ">debug g800" or whatever the command was?!
A possibility to set the priority for the init. task would be nice.
I am right now initializing a 1.4TB Volume.
The VMs that are running on the XenServer hosts are terribly slow now because of this.
I would not mind if the init. procedure would take twice as long or even longer if my VMs would work normally then..
It has nothing to do with CPU or RAM in my case. RAM usage is at 7% and CPUs are at normal temperature. BTW: is there a way to look at the CPU usage using the console? I can only see my CPU Temperatures using the IMPI module..
The init is quite fast -aprox. 1.5h for an 1.4TB volume on a Raid-6 (5x1TB discs on areca 1261 controller).
Dear Laxity,
THe CPU tempreture can be only seen from your IMPI card. But you can get other server stats from the check-Sys option in the system GUI. (STATUS_>Hardware)
Regards,
SJ
Thanks Sh-J,
I discovered the link to the statistics with your help :)
But the tool does not store history data as it seems.. When I click the button to display the values it always only gives me back the values of the last 15 minutes or so:
http://pic.ipicture.ru/uploads/081208/uFRS6y6ZBe.png
do I have to enable history logging somewhere?
Thanks in advance
Laxity
tried it again right now.. again only the data for the last 2 minutes is displayed..
any hints would be great! Thanks!
Dear Laxity,
I tested it on my server and I can see stats for long periods.
What release you are using?
Regards,
SJ
Hi Sh-J,
I am using Version: 5.0.DB49000000.3278
Regards,
Laxity
just a quick update:
after deleting the server statistics archive it all went fine.. so no problem here anymore..
Still a big fan of your software :-)
Regards, Laxity
A happy new year to you all!Quote:
Originally Posted by Laxity
And now to something complete different...
I've run an init of a 100GB FC volume yesterday... it took only a good handful of minutes, but caused a load of 26... all FC targets for our Xen server disconnected because of time-outs. I had quite a recovery job to do, a Sunday favorite of mine :-(
(We as well had no major CPU usage, memory was mostly free (we have 4GB in the machine, running in 32 bit mode with mostly FC targets that sure is a waste of memory), our RAID peaked at 190k sectors/s writes, which seems close to the maximum to be expected) And please don't blame the disks... as the disks are always the bottleneck, it's only a question of the FC volume size to bring the system to its limits with even the fastest disks. A proper system design would limit the priority of the init job to allow the productive resources to remain usable.
This is definitely a show-stopper for proper production use, we'll have to shut down all VMs prior to creating even mid-size (100GB nd above) FC targets :-(
With regards,
Jens
If i were you, I just would not bother to use file-io volumes. Stick with block-mode and suffer the performance penalty. This was my decision because I was trying to avoid this very same issue.
It's not unusual to make adds/remove to a SAN during business hours. The action should not cause a major service interruption..
JMO,
Good point that you make and a topic that we should add to our knowledgebase.
I agree that this will cause a performance hit due to the fact that it is using the dd command for the initialization. The best practice would be if you have a new system with 12TB (or whatever size) create the full size of 12TB (FC or iSCSI) Logical Volume and set it for the initialization. Once complete then you can delete it and you can create your logical volumes without. This will be the best way and know that your volume is good.
Other way would be to use the Volume Replication and failover to the destination server making it Source. Then once the init is completed failback, I know both can be time consuming vs the performance issue.
Tom can you guys figure out a different way of initializing volumes? What about nice-ing the dd command?
I will have to ask the engineers on this as there might be a trade off somewhere and I now that they had thought thru many variables.
Also there will have to be some test done to verify all before going into a release.
Update to initializing volumes w/ nice-ing. Engineers did try this with the past kernel 2.6.17 and it didn't work so well. They will look into with the latest kernel and if there are no issues or trade offs then you will see this in future releases.
Thanks guys for helping our engineers!
Just an update on the nice or ionice so you don’t think we forgot. Currently developers and are overloaded to take this subject back to research. We will ask them in a few months once freed up on pre existing projects. But they are going to look into it, just need more time.
What is the status of this Initialization being fixed to address the slow/unusable Open-e system when a 1.5TB LUN is being initialized?
Last night my Vsphere systems stopped responding due to an initialization. This problem creates OUTAGES, which in turn means Open-e has major problems.
Lets get a real answer Open-e engineers!!!
We addressed this with DSS V6 build 4221 and it is in the release notes. We give option to set Slow (uses less of the system resources), Medium and or Fast (uses more of the system resources - better to use when building a new system).
* File I/O iSCSI volumes and FC volumes no longer overload the system during initialization