I don’t usually take benchmarks very seriously. It’s worthwhile running them on new hardware as a quick check that everything’s working as expected - but if the results are within an order of magnitude of the hoped-for numbers after a single run, then I’m usually happy to move onto more productive tasks. Leave the endless tweaking and measurbation to the inhabitants of gentoo-land.
With flurry, though, I thought I should take a little more care. It has a 10 disk array, so the standard “ach, sure, raid 5 will do” instinct can be very dangerous. A single disk failure will leave the machine vulnerable for up to 72 hours - a couple of days to replace the disk, and another to rebuild the array. That’s a bit too long for comfort, especially if environmental factors have been the root cause of the initial failure.
So; I really, really wanted to go for RAID 6 - but I was unsure as to how much of a performance penalty that that would incur. My vague, handwave-y guess was that it’d be about a third slower in use, when compared to RAID 5. I’d consider 50% slower to be unacceptable, and anything less than 25% slower to be surprisingly good.
It turns out that bonnie++ was the best tool for the job. I was able to mimic the sort of operations that our current mail server does most often by using it with the following command line:
bonnie++ -b -d /home/rory/ -u rory -n 128:25000:500:16
ie. to write 128*1024 files to each of 16 directories, with a random variation of sizes between 500 and 25000 bytes (the average filesize on our current mailserver is 12.14 Kb - so that’s about right) - 25 Gigabytes of data in total. The -b option causes it to issue fsync() calls after each file has been written - again, this is the same setup that we’ll be using when the server goes live.
I ran that five times on a 1 TB “vanilla” ext3 partition (mounted noatime, like every other disk partition i’ve touched in the past five years!) sitting on top of a LVM volume, which in turn was mounted on the various types of RAID array (5, 6, 1+0, 0 [the latter for comedy value only, of course]) supported by out HP P400 card. I didn’t bother trying any form of software raid.
For comparison purposes, I also ran bonnie++ on a machine that is identical to ashes - which had served as a webserver from September 2000 to August 2005, and hasn’t been touched since then. It has a 30GB partition mounted at the start of the array (ashes has a 40GB one), which is formatted as reiserfs (as it is on ashes). It’s therefore going to give us a nice indication of how much (if at all) faster the new system is compared to what it’s replacing.
The results are as follows:
Well, unsurprisingly, all of the SATA RAID levels are faster than the old SCSI RAID 5 array for each of the four operations - between 13 and 14 times as fast for random reads (due mostly to having 10 spindles rather than just 4, I’m sure).
However, RAID 6 is “only” 3.01 times faster for random creates, compared to 5.87x for RAID5. That’s very close to being unacceptable to me, especially since this sort of operation accounts for almost a third of all those performed on our current mailserver.
Another option may be to go for RAID 5 + a hot spare. I’d end up with almost the same speed as the 10-disk RAID5 array, whilst being able to automatically being the array rebuild after the first failure - reducing the “danger time” by two thirds.
On the other hand, a mimimum of three times faster than the current system is still perfectly decent. I think I’m going to go need to do another round of measurebation aren’t I? Oh god, it’ll be -funroll-loops and buying a bass tube for my vauxhaul nova next…
