rryBlog

Two days left

Wed, 27 Aug 2008 18:57:00 BST

I finally leave work at the end of this week. Don’t ever let anyone try to convince you that signing a contract with a three month notice period is a good thing. Mm.

So, I’ve been “working from home” for the last couple of weeks. Hm, that makes it sound like a pleasant semi-holiday period, doesn’t it? And it’s true that the volume of work’s been reduced - no development work to support, and far fewer ops work now that we’ve transferred our public-facing services elsewhere, for a start. But, er, I’ll write a bit more about that at some stage in the future.

Anyway, the most productive thing I’ve done of late (apart from a quick spot of gallivanting) is to rip 343 cds to nice pristine FLAC format (and then transcoded from there to 128 and 256 kbps oggs for portability). Which is, er, about 100 more than I’ve managed to rip in the past eight months. At this rate, I might have my entire library finished at some stage in 2009. Go me, etc.

Anyway, to be written-about in the near future: the decline & fall of Sendit.com, the joys of job-hunting, and various techie bits (moving VMs from Xen to KVM to ESX and back again, observations on CPU vs I/O vs power consumption, the joy of having a properly-configured working environment, etc). All with an absolute minimum of emoicity. Hurrah!

344 CDs ripped now. Can I really own *that* much shoegaze-y stuff? Er, yes, it appears that I can. Oops.

Flurry rice

Fri, 17 Aug 2007 15:35:00 BST

I don’t usually take benchmarks very seriously. It’s worthwhile running them on new hardware as a quick check that everything’s working as expected - but if the results are within an order of magnitude of the hoped-for numbers after a single run, then I’m usually happy to move onto more productive tasks. Leave the endless tweaking and measurbation to the inhabitants of gentoo-land.

With flurry, though, I thought I should take a little more care. It has a 10 disk array, so the standard “ach, sure, raid 5 will do” instinct can be very dangerous. A single disk failure will leave the machine vulnerable for up to 72 hours - a couple of days to replace the disk, and another to rebuild the array. That’s a bit too long for comfort, especially if environmental factors have been the root cause of the initial failure.

So; I really, really wanted to go for RAID 6 - but I was unsure as to how much of a performance penalty that that would incur. My vague, handwave-y guess was that it’d be about a third slower in use, when compared to RAID 5. I’d consider 50% slower to be unacceptable, and anything less than 25% slower to be surprisingly good.

It turns out that bonnie++ was the best tool for the job. I was able to mimic the sort of operations that our current mail server does most often by using it with the following command line:
bonnie++ -b -d /home/rory/ -u rory -n 128:25000:500:16

ie. to write 128*1024 files to each of 16 directories, with a random variation of sizes between 500 and 25000 bytes (the average filesize on our current mailserver is 12.14 Kb - so that’s about right) - 25 Gigabytes of data in total. The -b option causes it to issue fsync() calls after each file has been written - again, this is the same setup that we’ll be using when the server goes live.

I ran that five times on a 1 TB “vanilla” ext3 partition (mounted noatime, like every other disk partition i’ve touched in the past five years!) sitting on top of a LVM volume, which in turn was mounted on the various types of RAID array (5, 6, 1+0, 0 [the latter for comedy value only, of course]) supported by out HP P400 card. I didn’t bother trying any form of software raid.

For comparison purposes, I also ran bonnie++ on a machine that is identical to ashes - which had served as a webserver from September 2000 to August 2005, and hasn’t been touched since then. It has a 30GB partition mounted at the start of the array (ashes has a 40GB one), which is formatted as reiserfs (as it is on ashes). It’s therefore going to give us a nice indication of how much (if at all) faster the new system is compared to what it’s replacing.

The results are as follows:

Well, unsurprisingly, all of the SATA RAID levels are faster than the old SCSI RAID 5 array for each of the four operations - between 13 and 14 times as fast for random reads (due mostly to having 10 spindles rather than just 4, I’m sure).

However, RAID 6 is “only” 3.01 times faster for random creates, compared to 5.87x for RAID5. That’s very close to being unacceptable to me, especially since this sort of operation accounts for almost a third of all those performed on our current mailserver.

Another option may be to go for RAID 5 + a hot spare. I’d end up with almost the same speed as the 10-disk RAID5 array, whilst being able to automatically being the array rebuild after the first failure - reducing the “danger time” by two thirds.

On the other hand, a mimimum of three times faster than the current system is still perfectly decent. I think I’m going to go need to do another round of measurebation aren’t I? Oh god, it’ll be -funroll-loops and buying a bass tube for my vauxhaul nova next…

Ashes to, er Flurry…

Tue, 14 Aug 2007 12:22:00 BST

Time for a new mailserver, then.

In an ideal world, I’d be putting the following on my shopping list:

HP DL380 G5, dual 2.67 GHz Xeon 5150 CPUs, 8 GB RAM, 8x 36 GB 2.5” 15k rpm SAS disks

- for the main mail server, anti-spam&virus, the mail queue (a large number of low-capacity 2.5” disks is the best route to achieving ultra-low seek times, which is important for randomly-accessed data like email), and sending our weekly mailshots.

HP DL320s, single 2.67 GHz Xeon 3070 CPU, 4 GB RAM, 12x 300 GB 3.5” 15k rpm SAS disks

- nfs server for users’ Maildirs, and the customer care mail database.

DL140 G3, dual 2.67 GHz Xeon 5150 CPUs, 8 GB RAM, 2x 100 GB 7.2 krpm SATA disks (for booting from only) x2

- user-facing servers - the first hosting a number of Xen instances for people to read mail using mutt or adjust their procmail setups, as well as pop3 and imap servers, and the second running webmail and the web frontend for the customer care mail system.

mm, tasty.

Unfortunately, I don’t have a budget of fifteen grand to spend on the above, so I’m going to make do with just the one box. In fact, it’s worse than that, as I’m also going to have to use this machine as a replacement for our ftp server, our “friends’n’family” webserver, and as a backup server (connected to a nice lto2 tape array).

As a result, I plumped for the following:

HP DL320s, single 2.67 GHz Xeon 3070 CPU, 4 GB RAM

- with an upgrade to a 512 MB battery-backed write cache

2x 72 GB 15k rpm 3.5” SAS disks

- a RAID 1 array for the mail queue and system partitions

10x 250 GB 7.2k rpm 3.5” SATA disks

- for users’ Maildirs

Total price was about £3,000 - a fifth of the cost of doing it right.

Meet flurry, our new mailserver

Things won’t be so awful for our technical staff - I’ll export their Maildirs to their own Xen instances on our big development server, laganside - so they can read their mail nice and quickly there. And I’ll probably inject the half-million message mailshots from infuse, a Xen instance elsewhere on our network. Even so - the new setup will merely provide a noticeable improvement to our users, rather tham being “zomg ultra-turbo-plus-plus!”. bah!

My task for the rest of week is to thoroughly benchmark the new machine, dubbed flurry. In particular, I’m interested to see the difference in speed between the various disk array setups that are open to me - JBOD (2.5 TB available), RAID 1+0 (1.25 TB available), RAID 5 (2.25 TB available), and RAID 6 (2.0 TB available).

My instinct is to go for either JBOD or RAID 6 - two disk failures will kill a RAID 5 array, and has a 50% chance of killing a RAID 1+0. With that number of disks, from the same manufacturer (a number of different batches, though), and subject to the same physical environment, the chances of experiencing multiple disk failures is higher than I’d like. I’m willing to be persuaded otherwise if the performance penalty for RAID 6 turns out to be huge, though.

Anyway, flurry has now been running memtest86+ for just over 24 hours, so it’s time for me to go start the benchmarking. hurrah!

Dust to dust, etc.

Tue, 14 Aug 2007 10:43:00 BST

I arrived at Sendit two-and-a-half years ago, and it quickly became obvious that every server needed to be replaced, and every system needed to be overhauled. That job is pretty much complete now - and, thanks to the magic of Xen, we’ve gone from having 60 or so servers to 11 (in fact, the savings in electricity alone will pay for the cost of the machines within the first half of their expected service life).

One of the last tasks on my list is to replace our mail server - something I’ve been looking forward to, as email is probably the closest I have to being a specialist subject. Our current server is ashes - a Dell Poweredge 2450 that entered service on 5th September, 2000. I’m aiming to do the switchover on its seventh birthday :)

Ashes has two 666 MHz Pentium III Xeon processors, a gigabyte of RAM, and 54 GB of disk (4x 18 GB 10krpm SCSI disks in a hardware RAID 5 array). It runs a fairly vanilla install of Qmail, qmail-popup for pop3, and courier-imap for imap4 (both of which are wrapped with stunnel for the ssl-ised variants). Some semblance of anti-spam measures are provided by rblsmtpd (pointed at the sbl-xbl.spamhaus.org blacklist), and most users also run spamassassin from their procmailrc. Anti-virus is provided by McAfee’s uvscan.

Our mail system has a couple of quirks - first, mail coming in to our customer care team is forwarded on to a separate server, angel (another PE2450, though slightly older) for entry into a mysql-back perl behemoth. Secondly, we send weekly special offers mails to half a million or so customers that have opted in to that service - this is done by a third server, y02, which is a shoddy *8* year old Dell Dimension XPS desktop box. eep!

Mail volume is pretty substantial - we have 91 “real” users, and 568 aliases (not counting all the various username-blah@ “dot-qmail”-style aliases). On a typical day, we would see around 350,000 delivery attempts, of which maybe 140,000 will be accepted into the system. Both of those figures can rise by 100,000 in the 24 hours after we send a mailshot, thanks to the staggering number of inventively-broken “out of office” autoresponders that our customers use.

This brings us to one of qmail’s major weaknesses. Since it doesnt check if a user actually exists before accepting mail in to the system, we can’t bounce backscatter / spam / improperly-addressed mail at SMTP time. Instead, we create around 40,000 new bounce messages every day, which might have been acceptable a decade ago, but is terribly anti-social these days.

Of the 100,000 emails that are delivered to users every day, perhaps three quarters are spam or viruses that either get sent to /dev/null or (all to often) end up in people’s inboxes. In short, more modern SMTP-time checks would save our server from doing an awful lot of work.

Finally, we have the issue of disk space. Ashes has 36 GB devoted to the /home partition. This has currently got less than a gigabyte free, and has never been less than 90% full since I’ve been here. Users have become adept at downloading mail, and storing it in whatever nasty, fragile binary format outlook uses. Even so, I have to harrass them every month or two to clean their inboxes out - a huge waste of everyone’s time. Since we use the dreaded RicerFS (in notail mode, too!) most Maildir/ files are unimaginably fragmented - to the point that opening a maildir containing 1,000 messages can take over two minutes.

Something needs to be done…

er, yes

Wed, 09 May 2007 20:08:00 BST

So, I’ve been meaning to move to a new blog system. Y’know, something shiney(tm). With bells and whistles and spam filtering. And, er, a billion or so critical security flaws uncovered each week.

In reality, I’m more than happy with blosxom. Vim-over-ssh is a much better way of writing entries than any editor written in AJAX-y javascript and hosted in a crash-prone web browser. And I don’t need on-the-fly spaling or grammar checks, ‘cos, well, I know how to write already, thanks.

So, yes, balls to wordpress, movable type and their overcomplex, php-driven, sql db-backed ilk.

So, er, what have I been doing in the last 171 days?

Well, travelling lots: Firstly, swanning around Ofotfjord and Vestjord in the far north of Norway, supposedly to see orcas.

Shitty picture as we were on a boat, and the sun didnt ever get above the horizon

Then, a week spent in snowy Tallinn, in Estonia. I’m not sure what I was expecting from part of the former Soviet Union - but I was shocked by how prosperous, clean, and unspoilt it was. The walled old town is an astonishingly pretty medieval fairy story brought to life; every restaurant seems to offer bearmeat and wild boar and serving wenches (Olde Hansa scores several billion bonus points for the post-meal music, provided corsetted students from the local university’s medieval music department). I’m fairly sure I’ll be back in the future…

The town hall, built in 1371, just round the corner from our apartment

I’ve also spent time in Leeds and Donegal, and met interesting people, and done interesting things. But, er, I’m flagging a bit, and I’m sure you don’t really care. fleh. Will write more soon :)

er, yes

Sun, 19 Nov 2006 15:04:00 BST

So, I’ve been meaning to move to a new blog system. Y’know, something shiney(tm). With bells and whistles and spam filtering. And, er, a billion or so critical security flaws uncovered each week.

In reality, I’m more than happy with blosxom. Vim-over-ssh is a much better way of writing entries than any editor written in AJAX-y javascript and hosted in a crash-prone web browser. And I don’t need on-the-fly spaling or grammar checks, ‘cos, well, I know how to write already, thanks.

So, yes, balls to <a href="http://wordpress.org/development/2007/03/upgrade-212/">wordpress</a>, <a href="http://www.sixapart.com/movabletype/pricing">movable type</a> and their overcomplex, php-driven, sql db-backed ilk.

So, er, what have I been doing in the last 171 days?

Well, travelling lots: Firstly, swanning around <a href="http://en.wikipedia.org/wiki/Ofotfjord">Ofotfjord</a> and <a href="http://en.wikipedia.org/wiki/Vestfjord">Vestjord</a> in the far north of Norway, supposedly to see orcas. <center><img src="http://klub.org/~rory/tysfjord.jpg" border=2> Shitty picture as we were on a boat, and the sun didnt ever get above the horizon</center>

Then, a week spent in snowy <a href="http://wikitravel.org/en/Tallinn">Tallinn</a>, in Estonia. I’m not sure what I was expecting from part of the former Soviet Union - but I was shocked by how prosperous, clean, and unspoilt it was. The walled old town is an astonishingly pretty medieval fairy story brought to life; every restaurant seems to offer bearmeat and wild boar and serving wenches (<a href="http://www.oldehansa.ee/?lang=en&id=1070&tpl=1002">Olde Hansa</a> scores several billion bonus points for the post-meal music, provided corsetted students from the local university’s medieval music department). I’m fairly sure I’ll be back in the future…

<center> <img src="http://klub.org/~rory/raekojaplats.jpg" border="2"> The town hall, built in 1371, just round the corner from our apartment</center>

I’ve also spent time in Leeds and Donegal, and met interesting people, and done interesting things. But, er, I’m flagging a bit, and I’m sure you don’t really care. fleh. Will write more soon :)

er, yes

Sun, 19 Nov 2006 15:03:00 BST

So, I&#8217;ve been meaning to move to a new blog system. Y&#8217;know, something shiney(tm). With bells and whistles and spam filtering. And, er, a billion or so critical security flaws uncovered each week.

In reality, I&#8217;m more than happy with blosxom. Vim-over-ssh is a much better way of writing entries than any editor written in AJAX-y javascript and hosted in a crash-prone web browser. And I don&#8217;t need on-the-fly spaling or grammar checks, &#8216;cos, well, I know how to write already, thanks.

So, yes, balls to <a href="http://wordpress.org/development/2007/03/upgrade-212/">wordpress</a>, <a href="http://www.sixapart.com/movabletype/pricing">movable type</a> and their overcomplex, php-driven, sql db-backed ilk.

So, er, what have I been doing in the last 171 days?

Well, travelling lots: Firstly, swanning around <a href="http://en.wikipedia.org/wiki/Ofotfjord">Ofotfjord</a> and <a href="http://en.wikipedia.org/wiki/Vestfjord">Vestjord</a> in the far north of Norway, supposedly to see orcas. <center><img src="http://klub.org/~rory/tysfjord.jpg" border=2> Shitty picture as we were on a boat, and the sun didnt ever get above the horizon</center>

Then, a week spent in snowy <a href="http://wikitravel.org/en/Tallinn">Tallinn</a>, in Estonia. I&#8217;m not sure what I was expecting from part of the former Soviet Union - but I was shocked by how prosperous, clean, and unspoilt it was. The walled old town is an astonishingly pretty medieval fairy story brought to life; every restaurant seems to offer bearmeat and wild boar and serving wenches (<a href="http://www.oldehansa.ee/?lang=en&id=1070&tpl=1002">Olde Hansa</a> scores several billion bonus points for the post-meal music, provided corsetted students from the local university&#8217;s medieval music department). I&#8217;m fairly sure I&#8217;ll be back in the future&#8230;

<center> <img src="http://klub.org/~rory/raekojaplats.jpg" border="2"> The town hall, built in 1371, just round the corner from our apartment</center>

I&#8217;ve also spent time in Leeds and Donegal, and met interesting people, and done interesting things. But, er, I&#8217;m flagging a bit, and I&#8217;m sure you don&#8217;t really care. fleh. Will write more soon :)

er, yes

Wed, 18 Oct 2006 17:21:00 BST

&lt;p class=&quot;blogtext&quot;&gt; So, I&amp;#8217;ve been meaning to move to a new blog system. Y&amp;#8217;know, something shiney(tm). With bells and whistles and spam filtering. And, er, a billion or so critical security flaws uncovered each week. &lt;/p&gt;

&lt;p class=&quot;blogtext&quot;&gt; In reality, I&amp;#8217;m more than happy with blosxom. Vim-over-ssh is a much better way of writing entries than any editor written in AJAX-y javascript and hosted in a crash-prone web browser. And I don&amp;#8217;t need on-the-fly spaling or grammar checks, &amp;#8216;cos, well, I know how to write already, thanks. &lt;/p&gt;

&lt;p class=&quot;blogtext&quot;&gt; So, yes, balls to &lt;a href=&quot;http://wordpress.org/development/2007/03/upgrade-212/&quot;&gt;wordpress&lt;/a&gt;, &lt;a href=&quot;http://www.sixapart.com/movabletype/pricing&quot;&gt;movable type&lt;/a&gt; and their overcomplex, php-driven, sql db-backed ilk. &lt;/p&gt;

&lt;p class=&quot;blogtext&quot;&gt; So, er, what have I been doing in the last 171 days? &lt;/p&gt;

&lt;p class=&quot;blogtext&quot;&gt; Well, travelling lots: Firstly, swanning around &lt;a href=&quot;http://en.wikipedia.org/wiki/Ofotfjord&quot;&gt;Ofotfjord&lt;/a&gt; and &lt;a href=&quot;http://en.wikipedia.org/wiki/Vestfjord&quot;&gt;Vestjord&lt;/a&gt; in the far north of Norway, supposedly to see orcas. &lt;center&gt;&lt;img src=&quot;http://klub.org/~rory/tysfjord.jpg&quot; border=2&gt;&lt;br&gt;&lt;i&gt;Shitty picture as we were on a boat, and the sun didnt ever get above the horizon&lt;/i&gt;&lt;/center&gt;

&lt;p class=&quot;blogtext&quot;&gt; Then, a week spent in snowy &lt;a href=&quot;http://wikitravel.org/en/Tallinn&quot;&gt;Tallinn&lt;/a&gt;, in Estonia. I&amp;#8217;m not sure what I was expecting from part of the former Soviet Union - but I was shocked by how prosperous, clean, and unspoilt it was. The walled old town is an astonishingly pretty medieval fairy story brought to life; every restaurant seems to offer bearmeat and wild boar and serving wenches (&lt;a href=&quot;http://www.oldehansa.ee/?lang=en&amp;id=1070&amp;tpl=1002&quot;&gt;Olde Hansa&lt;/a&gt; scores several billion bonus points for the post-meal music, provided corsetted students from the local university&amp;#8217;s medieval music department). I&amp;#8217;m fairly sure I&amp;#8217;ll be back in the future&amp;#8230; &lt;/p&gt;

&lt;center&gt; &lt;img src=&quot;http://klub.org/~rory/raekojaplats.jpg&quot; border=&quot;2&quot;&gt;&lt;br&gt;&lt;i&gt;The town hall, built in 1371, just round the corner from our apartment&lt;/i&gt;&lt;/center&gt;

&lt;p class=&quot;blogtext&quot;&gt; I&amp;#8217;ve also spent time in Leeds and Donegal, and met interesting people, and done interesting things. But, er, I&amp;#8217;m flagging a bit, and I&amp;#8217;m sure you don&amp;#8217;t really care. fleh. Will write more soon :) &lt;/p&gt;