I don’t usually take benchmarks very seriously. It’s worthwhile running them on new hardware as a quick check that everything’s working as expected - but if the results are within an order of magnitude of the hoped-for numbers after a single run, then I’m usually happy to move onto more productive tasks. Leave the endless tweaking and measurbation to the inhabitants of gentoo-land.

With flurry, though, I thought I should take a little more care. It has a 10 disk array, so the standard “ach, sure, raid 5 will do” instinct can be very dangerous. A single disk failure will leave the machine vulnerable for up to 72 hours - a couple of days to replace the disk, and another to rebuild the array. That’s a bit too long for comfort, especially if environmental factors have been the root cause of the initial failure.

So; I really, really wanted to go for RAID 6 - but I was unsure as to how much of a performance penalty that that would incur. My vague, handwave-y guess was that it’d be about a third slower in use, when compared to RAID 5. I’d consider 50% slower to be unacceptable, and anything less than 25% slower to be surprisingly good.

Time for a new mailserver, then.

In an ideal world, I’d be putting the following on my shopping list:

I arrived at Sendit two-and-a-half years ago, and it quickly became obvious that every server needed to be replaced, and every system needed to be overhauled. That job is pretty much complete now - and, thanks to the magic of Xen, we’ve gone from having 60 or so servers to 11 (in fact, the savings in electricity alone will pay for the cost of the machines within the first half of their expected service life).

One of the last tasks on my list is to replace our mail server - something I’ve been looking forward to, as email is probably the closest I have to being a specialist subject. Our current server is ashes - a Dell Poweredge 2450 that entered service on 5th September, 2000. I’m aiming to do the switchover on its seventh birthday :)

Ashes has two 666 MHz Pentium III Xeon processors, a gigabyte of RAM, and 54 GB of disk (4x 18 GB 10krpm SCSI disks in a hardware RAID 5 array). It runs a fairly vanilla install of Qmail, qmail-popup for pop3, and courier-imap for imap4 (both of which are wrapped with stunnel for the ssl-ised variants). Some semblance of anti-spam measures are provided by rblsmtpd (pointed at the sbl-xbl.spamhaus.org blacklist), and most users also run spamassassin from their procmailrc. Anti-virus is provided by McAfee’s uvscan.

Our mail system has a couple of quirks - first, mail coming in to our customer care team is forwarded on to a separate server, angel (another PE2450, though slightly older) for entry into a mysql-back perl behemoth. Secondly, we send weekly special offers mails to half a million or so customers that have opted in to that service - this is done by a third server, y02, which is a shoddy *8* year old Dell Dimension XPS desktop box. eep!

Mail volume is pretty substantial - we have 91 “real” users, and 568 aliases (not counting all the various username-blah@ “dot-qmail”-style aliases). On a typical day, we would see around 350,000 delivery attempts, of which maybe 140,000 will be accepted into the system. Both of those figures can rise by 100,000 in the 24 hours after we send a mailshot, thanks to the staggering number of inventively-broken “out of office” autoresponders that our customers use.

This brings us to one of qmail’s major weaknesses. Since it doesnt check if a user actually exists before accepting mail in to the system, we can’t bounce backscatter / spam / improperly-addressed mail at SMTP time. Instead, we create around 40,000 new bounce messages every day, which might have been acceptable a decade ago, but is terribly anti-social these days.

Of the 100,000 emails that are delivered to users every day, perhaps three quarters are spam or viruses that either get sent to /dev/null or (all to often) end up in people’s inboxes. In short, more modern SMTP-time checks would save our server from doing an awful lot of work.

Finally, we have the issue of disk space. Ashes has 36 GB devoted to the /home partition. This has currently got less than a gigabyte free, and has never been less than 90% full since I’ve been here. Users have become adept at downloading mail, and storing it in whatever nasty, fragile binary format outlook uses. Even so, I have to harrass them every month or two to clean their inboxes out - a huge waste of everyone’s time. Since we use the dreaded RicerFS (in notail mode, too!) most Maildir/ files are unimaginably fragmented - to the point that opening a maildir containing 1,000 messages can take over two minutes.

Something needs to be done…

Well, my home network does, at least. Marrow is my 1 week old media storage server, and has almost a terabyte of storage space - which is exported via nfs to the other machines in my house.

The machine itself is an old Dell Dimension XPS. The “XPS” is Dell-speak for “not quite as shit as a standard Dimension”. I’ve always been a fan of Dell Optiplex and Precision desktops - but the Dimension and XPS series make the Slashdot crowd’s sneery attitude towards Dell desktops seem justified. ah well.

Anyway, when bought, marrow had a 700 MHz Pentium 3 cpu, 256 MB RAM, 20 GB of hard disks, and a quad speed cd-rom drive. Now, it has er, a 700 MHz cpu, 756 MB RAM, 5x 250MB disks, 2x five disc cd changers, and a dvd-rw drive. Anyone would think that it’d been pilfering from a cyclist’s medicine cabinet.

Each disk has a two partitions on it, of 5 GB and 245 GB:

+------------++------------++- - - - - - +   +------------------------+
|            ||            |             |   |                        |
| /dev/hda1  || /dev/hdb1  || /dev/hdc1      | /dev/hdd1    /dev/hde1 |
| raid 1     || raid 1     |  raid 1     |   | raid 0                 |
| / (mirror) || / (mirror) || / (spare)      | swap                   |
|            ||            |             |   |                        |
+------------++------------++ - - - - - -+   +------------------------+

+---------------------------------------------------------------------+
|                                                                     |
| /dev/hda2     /dev/hdb2      /dev/hdc2       /dev/hdd2    /dev/hde2 |
| raid 5                                                              |
| mounted on /srv                                                     |
|                                                                     |
+---------------------------------------------------------------------+

So, we have a redundant, mirrored 5 GB root partition, with a hot spare. 10 Gb of swap is available on striped set - this is greeat for a toy setup, but a silly idea for a real machine, as failure of either disk will cause the machine to die spectacularly if/when it tries to use more than half of the available swap space.

I’ve been fiddling around to see the best parameters to use with nfs. After lots of geeky testing, I’ve come up with the following mount line:

marrow:/srv/music /home/rory/music nfs rw,noatime,rsize=32768,wsize=32768,hard,intr

noatime: Don’t update the file’s last access timestamp - I’ve never used the atime for anything, so I’m happy not to have to to the remote server every time I touch a file.

rsize, wsize: These values surprised me. The default is 2048, and the standard recommendation is to increase it to 8192 if your server supports large values. To be honest, I’d expected that 64k - the server’s raid 5 stripe size - would provide the best performance; but, in my tests, 32k was significantly faster than any other value (with 16k being marginally faster than either 8k or 64k). I have a feeling that this is because the mode of the file sizes in my trial set (my homedir, essentially) was 31.5k. In other words, the optimal size of the rsize and wsize values are dependant on the size of the files being used.

hard: This allows file operations to be retried even if they initially time out - which means that the nfs server can go away for a reboot without causing any client-side problems

intr: hard mounting has a drawback, however - if the server crashes, the client will keep on trying to access it. You won’t be able to close any program that has a lock on a remote file, or unmount any remote partitions. intr fixes this by letting the client interrupt any such operation. hurrah!

Finally, a note about using nfs on a desktop system. Desktop enivronments, like GNOME, need to know whats happening to files in a users homedir, so that they can show those changes in the file manager, etc.

The obvious way to do this is to periodically poll each file for changes. This is, of course, slow, system intensive, and icky. A much better way is for the kernel to notify the desktop each time that some other subsystem changes a file - something it does using the older dnotify, or shiny new inotify interface.

In the olden days, we used “fam” to listen for dnotify events. You could run it on a remote nfs server, and get it to broadcast update notifications to nfs client machines, which was all very handy. However, fam has now been replaced with “gamin”, which is more lightweight (and so less system-intensive), and can listen for inotify events rather than dnotify (again, less system-intensive).

Unfortunately, gamin can’t do the funky listen-to-a-remote-fam-server thing. This is supposedly a feature-not-a-bug. “security, donchaknow”. BUT, JESUS FUCK, NFS IS WILDLY INSECURE /ANYWAY/. cretins.

So, gamin falls back to polling all your nfs mounts every 10 seconds. Braindead, yes? So, ease the pain by making it every 90s instead, by editing your /etc/gamin/gaminrc to add the following line:

fsset nfs poll 90

With that added, marrow is able to saturate the network to both humble (full duplex eth100) and sanguine (802.11g) - which is exactly what I wanted to achieve. hurrah!

Sarah’s been busy in the studio, taking photos of various gorgeous people for her project on body modification. Yesterday’s session produced around a gigabyte of images, which were uploaded to wintermute for safekeeping.

Unfortunately, some of them didn’t turn out properly. In this one, for example, the studio flash didn’t fire - so the image was taken with the modelling lights only. Therefore, it’s underexposed and the colour balance is badly off.

The four six-year-old Dell Poweredge 2300 machines that we’d be running our main site on have slowly been dying off over the last few months.

In July, I bought two brand new Poweredge 1850s as replacements. With dual Xeon64 processors and a couple of gigs of ram, they should be more than adequate, alongside a couple of the older boxes. I cloned an image of the old server setup onto one of them, and put it into service immediately. I was intending to install Debian Sarge from scratch onto the other, but various perl module incompatibilities need more care and attention than I’ve had time to give of late.

Yesterday, though, another of the old servers failed. We’re also running a very successful promotion at the moment, which has increased pressure on our machines. Last week, the PE1850, carrying 50% of traffic, was experiencing a load avg of around 0.4. Yesterday evening, carrying 75% of traffic, it was seeing a load of around 9.5. With dual hyper-threaded processors, a load average of more than 4 will mean that customers may notice slowdowns in accessing the site.

So; I decided to give up on the install-from-scratch, and simply clone the server system image onto the second PE1850. That took about six hours in all - there’s a 100 Mbps LES line between the datacentre and my office, but only a 10 Mbps hub at my desk (beh!).

Strangely, though, I couldn’t get the second new Poweredge to show all available memory in Linux. I tried Linux 2.4 and 2.6 kernels, and both the lilo and grub bootloaders, but saw only 256 MB ram with each. agh!

Both have the same kernel config - with CONFIG_MPENTIUM4, CONFIG_HIGHMEM4G and CONFIG_HIGHMEM all set (and CONFIG_NOHIGHMEM unset, obviously). I ran a diff between the dmesg output on both machines, and the only difference was a “allocated 32 pages and 32 bhs reserved for the highmem bounces” line on the “working” machine that wasn’t present on the broken one. That’s diagnostic, though - not anything that would /cause/ problems. And, yes, I’d tried passing “mem=2048M” on the kernel’s command line.

Strangely, all two gigs of RAM could be seen in the bios’ memory test, and running memtest86, both from the boot loader and from the commandline.

So, eventually, I went to have a hoke through the BIOS settings. I hadnt changed either of them from the factory defaults, but it seems that the “broken” machine has “OS Install mode” selected, whereas the other didn’t. What does “OS Install mode” do? Why, it limits the amount of memory reported to 256 MB. agh!

I turned that off, and lo, all 2048MB was visible in linux! hurrah! But, why had that been set in the first place? Who knows? And what possible use can it have? I’m presuming it’s something to do with OS/2 or Windows NT4 compatibility, but even so… grrrrrr!

…but even when it was working, it had never done a very good job on my living room carpet, which is exceptionally cheap and nasty. It’s woven from artificial fibres, which moult and then get stuck in the remaining pile, forming hairballs. The hoover wasn’t powerful enough to do more than push those around; it functioned more as an old-fashioned carpet sweeper, and left the place looking almost as filthy as if I didn’t bother to hoover at all.

So; I needed a new hoover. Immediately discounting the overpriced dyson-type tat, I went to look for one of those “henry” devices that office cleaners use. I reasoned that they cost about £25, so were essentially disposable. Better still - though they may not have nilfisk-grade filtration, or any other “features” to speak of, they are powerful and reliable enough for industrial use, and are therefore easily good enough for use in the not.nothovel.

Unfortunately, that idea seems to have occurred to more than a few other people, and Henry prices have quadrupled in the last five years. bah.