Spinning Yarns

Eventually, I concluded that my recent experiment with switching to Mail.app was a failure. So, I installed a local copy of Mutt to read with, Fetchmail to keep my in-box populated, and Maildrop to help filter out the spam. Thanks to the magic of MacPorts, this was all fairly simple to do—however, there were a few subtle tricks to getting it set up, so I thought I’d record how I did it, in case anybody else might benefit from the effort.

For reference, as of this writing I’m using MacOS 10.4.11 (“Tiger”) and the Fetchmail 6.3.8 that comes pre-installed from Apple. Your mileage may vary if your configuration differs, though it is likely to work similarly with 10.5 (“Leopard”).

Contents:

• Installing Mutt and Maildrop
• Configuring Mutt and Maildrop
• Configuring Fetchmail
• Converting Mailboxes

Installing Mutt and Maildrop

I wanted a version of Mutt that could handle IMAP/SSL and POP3/SSL for incoming mail, and SMTP with STARTTLS for outgoing mail. Thanks to the lovely MacPorts ‘variants’ system, this was easy:

% sudo port install mutt-devel +imap +pop +smtp +sasl +ssl

This will pull in a few other packages, including expat, gdbm, gperf, and ncurses, but nothing that takes too long to build. You may wonder why I would mention build time—let me just say you should be glad you haven’t had to compile ghc lately.

Maildrop is the mail delivery agent from the Courier MTA, but it is also available as a stand-alone tool. It has a more graceful configuration syntax than my old standby Procmail, and since my filter setup is a lot simpler now than it used to be, I decided to make the switch. Once again, an easy install from MacPorts,

% sudo port install maildrop

This will pull in pcre, but I think that’s it. Some of the examples that follow assume that maildrop is installed as /opt/local/bin/maildrop, but you can edit the pathname accordingly as needed. You don’t need to install fetchmail, since it comes pre-installed as /usr/bin/fetchmail.

Configuring Mutt and Maildrop

After many years of corrupted mbox files, I decided now was the time to switch to using Maildir instead. The Maildrop package has a tool called maildirmake that makes it easy to get things set up, which I did as follows:

% maildirmake ~/Documents/Maildir

The common practise for Maildir is that nested folders are all stored at the top level of the directory, with names starting with a period (.) and nesting to be indicated by dots. However, as long as you are not trying to use your Maildir folder with Courier, it works just fine to create nested folders directly using maildirmake, e.g.,

% maildirmake ~/Documents/Maildir/Spam
% maildirmake ~/Documents/Maildir/Friends

Although nesting folders this way technically violates the Maildir spec, Mutt and Maildrop understand it just fine. Telling Mutt to use the new format required only a couple of lines in ~/.muttrc, namely:

set folder = "~/Documents/Maildir"
set mbox_type = Maildir

There are some other useful pointers for using Mutt with Maildir, but this is all that’s really necessary.

Configuring Fetchmail

Configuring Fetchmail to download my e-mail and to play nicely with the MacOS launchd tools was probably the hardest part of this process. Fortunately, it can be stripped down to a few simple instructions:

1. Create a property list file for launchd. The easiest way to do this is to open the Property List Editor application,* but you can write it by hand, if you prefer. Mine looks like this:

   < ?xml version="1.0" encoding="UTF-8"?>
   < !DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" 
      "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
   <plist version="1.0">
   <dict>
   	<key>Disabled</key>
   	<false />
   	<key>Label</key>
   	<string>local.fetchmail</string>
   	<key>ProgramArguments</key>
   	<array>
   		<string>/Users/michael/scripts/fm.sh</string>
   		<string>--syslog</string>
   		<string>--nodetach</string>
   	</array>
   	<key>RunAtLoad</key>
   	<true />
   	<key>StartInterval</key>
   	<integer>240</integer>
   </dict>
   </plist>

   This configuration will tell the launchd tool to run fetchmail periodically to check for new mail. The Label field is a string that describes the job; I just chose local.fetchmail as being suitably descriptive. The StartInterval tells it how often to run—in my case, I have it running every 4 minutes (i.e., 240 seconds). The ProgramArguments array tells it what program to run, and what command-line arguments it should receive. The one that matters most here is --nodetach (or -N for short), which prevents fetchmail from going into the background as a daemon process. You can let it run that way, but it doesn’t play nicely with launchd.

   Once you have this file filled in to suit your own desires, save it in ~/Library/LaunchAgents/local.fetchmail.plist. You may need to create the ~/Library/LaunchAgents/ directory, but there’s nothing special about it; just make a new folder in the Finder if it’s missing. Storing the property list here will permit launchd to find it when you log in. You can get more information about launchd from Apple’s web site, if you’re interested.

   Note:

   You could use cron instead of launchd if you prefer; I did not do so because cron jobs run all the time, and I wanted something that would run only when I’m actually logged in using the machine. If you want to use cron, I assume you’re probably savvy enough to figure out how to make it work.

2. Install a wrapper for fetchmail. Notice that I am not running fetchmail directly, but instead am running a shell script named fm.sh. The reason for this is that when fetchmail discovers that you have no messages waiting, it exits with a status code of 1. By convention, any status code other than zero is interpreted as an “error” of some kind, and so launchd would get confused and think that fetchmail had failed. If it fails too many times, launchd gives up and stops running it. To work around this, my script just runs fetchmail, and if it returns status code 1, my script returns status code 0. This keeps launchd happy, without changing the behavior of the program. Here’s what that script looks like:

   #!/bin/sh 
   fetchmail $*
   fmstat=$?
   if [ $fmstat = 1 ] ; then
      exit 0
   else
      if [ $fmstat = 75 ] ; then
        growlnotify --sticky --message "An e-mail was not filtered correctly."
      elif [ $fmstat != 0 ] ; then
        syslog -s -l NOTICE "fm.sh: fetchmail exited with status ($fmstat)"
      fi
      exit $fmstat
   fi
   # Here there be dragons

   You will need to make sure the script is executable, and of course you should modify the path in the property list to point to wherever you actually keep it.

3. Set up your fetchmail configuration. How to do this depends strongly upon where you get your mail from. The important points are that (a) The configuration should be stored in a text file named ~/.fetchmailrc, and (b) That file should not be readable or writable by anyone but you.

   I generally fetch my e-mail from a handful of different IMAP/SSL servers, so my .fetchmailrc file looks similar to this (except, of course, that none of my login names or passwords is actually shown here in the example)

   
   set invisible # don't modify message headers
   
   poll imap.domain1.com
     with protocol IMAP;
     user "login1" with password "secret1" is "yourname" here,
       and wants mda "/opt/local/bin/maildrop"
     nokeep nofetchall
     sslfingerprint
      "01:02:03:04:05:06:07:08:09:0A:0B:0C:0D:0E:0F:10"
   
   poll maildrop.domain2.org
     with protocol POP3;
     user "login1"
     with password "secret2" is "yourname" here,
       and wants mda "/opt/local/bin/maildrop"
     nokeep nofetchall sslproto ssl23
     sslfingerprint
      "11:12:13:14:15:16:17:18:19:1A:1B:1C:1D:1E:1F:20"
   
   poll pop.gmail.com
     with protocol POP3;
     user "your.account@gmail.com" there
     with password "secret3" is "yourname" here,
       and wants mda "/opt/local/bin/maildrop"
     nokeep nofetchall ssl
     sslfingerprint
      "87:47:EA:7F:C6:B2:D1:DB:D3:64:4C:23:17:DB:E3:05"
   

   You will need to substitute in the appropriate host names, login names, and passwords. In the example, “yourname” means your user-name on the local machine, where you are planning to read mail once it’s delivered, while “login1” and “secret1” mean your user-name and secret password for your e-mail account on the remote server. The third example uses GMail, so in that case you could actually leave the host name alone, provided you’ve modified your GMail account settings to allow POP3 access. Check out the fetchmail(1) manual page if you have other needs.

   The nokeep option tells fetchmail to delete the messages from the server once it is finished downloading; if you don’t like that, use keep in its place, which instructs it to leave your messages on the server—they will be skipped over for subsequent downloads. Using keep is probably a good idea while you are testing your configuration. The sslfingerprint options can be omitted, but I have included them to keep fetchmail from complaining about the self-signed server certificates many mail hosts use in lieu of a properly-signed certificate.**

4. Set up your maildrop configuration. If you just want to drop mail into your default mailbox, this is easy: Just create a textfile named ~/.mailfilter that contains something similar to this:

   
   DEFAULT=$HOME/Documents/Maildir
   

   Really and truly, that’s all you need; everything you don’t otherwise filter will be delivered to your default mailbox. My configuration is a tiny bit more complex, because I have a spam filter that runs on each incoming message, and messages it flags as spam are diverted to a separate mailbox. I also keep around a backup of the last 50 non-spam messages I received, in case I do something stupid and delete one—the recipe for that I lifted right out of the maildrop documentations. For reference, here’s how my configuration looks:

   
   DEFAULT="$HOME/Documents/Maildir"
   PATH="$PATH:/path/to/my/programs"
   
   xfilter "mailtagger" # run the spam filter
   
   # Check against spam filter
   if ( /^X-MailTag: spam/:h )
     to "$DEFAULT/Spam"
   
   # Keep a backup of the last several messages delivered.
   # Recipe taken from maildropex(7) manual page.
   cc "$DEFAULT/XBackup/"
   `cd "$DEFAULT/XBackup/new" && rm -f dummy \`ls -t | sed -e 1,50d\``
   
   # Here there be dragons
   

At this point, you should be good to go. You can test your fetchmail configuration by running it a couple of times from the command line, e.g.,

% fetchmail -v -v --nodetach

Once you’re satisfied, it’s time to hand off the process to launchd, for which you should run:

% launchctl load ~/Library/LaunchAgents/local.fetchmail.plist

Modify accordingly if you saved your property list from Step (1) under a different name. At this point, Fetchmail should be fetching your e-mail from whatever servers you specified. I recommend you send yourself a couple of messages just to make sure it’s all right. Errors from fetchmail are written in the console log, so if you’re not getting what you expected, you should look there for feedback about why.

Converting Mailboxes

If all your mail resides on the mail server, you should be all set; however if you have any e-mail already saved locally under the control of Mail.app, you will have to do a bit of work to convert it.

Mail.app stores your mail in ~/Library/Mail/Mailboxes/, using a format similar to Maildir. If it were actually Maildir, you could use it directly, but it’s not quite the same. Each mailbox is stored in a directory named whatever.mbox, which contains an Info.plist that record some user preferences. However, instead of using three subdirectories (cur, new, and tmp) per mailbox, Mail.app uses only a single directory named Messages for each of its mail folders. Within Messages, each message is stored in a text file named something like nnnnnn.emlx, where nnnnnn is a sequence number of some kind; as far as I can tell, the numbers contain no information other than their identity.

The first line of an .emlx file contains a decimal integer, specifying the number n of bytes in the message. The next n bytes after that first line contain the message data itself, and the remainder of the file contains a short XML property list giving some attributes of the message. For my purposes, the easiest solution is to just discard the byte count and the property list entirely. The following Python code would suffice to convert a single .emlx file src and write the result into a new file dst:

def unpack_emlx(src, dst):
    with file(src, 'rb') as ifp:
        dlen = int(ifp.readline())
        data = ifp.read(dlen)
 
    with file(dst, 'wb') as ofp:
        ofp.write(data)

To satisfy Maildir, you need simply extract each message into a uniquely-named file, and then move those extracted messages into your Maildir’s new folder. Once you have converted each of the .emlx files, the rest can be accomplished using mv. It is a tedious process, but not difficult. The unpack_emlx() function shown here is part of a somewhat longer script I wrote that applies it to each .emlx file found in a user-specified list of directories. I won’t post the code here, since it’s not really polished enough for others to use it, but if you really want a copy, drop me a line and I can file off some of its rough edges for you.

^* This might not be available if you haven’t installed the developer tools, but on my system it can be found in
/Developer/Applications/Utilities.

^** A rant about the foolish overcomplexity of the host certificate validation process is probably best saved for another day.

Filed by Michael at 17:21 under Technology, Tutorial
No Comments

E-mail comprises an enormous fraction of my daily communication with others. For some situations, that turns out to be a blessing, but generally speaking it’s just one of those compromises I have to live with since I can’t be everywhere at once. As a result, however, I am somewhat picky about the tools I use to read, write, send, and receive e-mail messages, and I am a member of a rapidly dwindling minority of Internet users who really dislike the tendency of modern e-mail software to confuse “writing e-mail” with “word processing.” In my view, an e-mail message should consist primarily of plain, unformatted text. Not HTML. Not PDF. Not XML. Not OpenDoc, or Microsoft Word, or RTF. Not pictures, sound-files, movies, animations, plug-ins, or widgets. Plain, unformatted text.

Lest you think me a Luddite, I should point out that I’m not insisting on ASCII text. I’m perfectly happy to receive messages written in Unicode—in fact, I prefer it. But any styles or formatting other than line breaks should be reserved for some other medium, as they have no place in e-mail. As I write this, I realize my attitude is considered antediluvean, and will eventually be swept away by the rising bilge of markup languages, but I stand by my conservatism on this point with whole heart.

Unfortunately, the only tools that support this somewhat crotchety viewpoint tend to be grumpy and ill-maintained terminal-based programs that must be manually installed and configured, and whose upkeep is a task best described as Herculean (think “Augean stables” here). Having done this for many years now, I decided maybe it was time to give the graphical programs another chance. So, when it came time to move my e-mail off the servers at Dartmouth, I decided to see whether I could live with the Mail application that Apple provides with their operating system. So far, I’m not very impressed.

So, the thing about GUI applications, as compared with their text-based command-line counterparts, is that it’s generally much easier to get started with the GUI program. You boot the thing up, and right away you can see all its essential features. You can look through the menus to see what commands it provides; bring up windows and dialog boxes to control its operating parameters; and in many cases, even browse help files within the application. You don’t have to memorize a bunch of weird key-commands before you can accomplish anything, and for the most part, the default settings are good enough to get the average user up and running. Text-driven software is rarely this easy for the first-time user, or even the second-time user.

On the other hand, as soon as you get past that initial learning curve, and you want to exert some control over what your software is doing, the GUI program makes everything about fifteen times harder than it should be. Okay, so your text-mode e-mail client made you edit an arcane configuration file and learn a bunch of keyboard commands before you could even view your In Box. But now you want to change how your messages are displayed? No problem! Just edit that textfile a couple more times, and it’ll do your bidding. You want to do that with your GUI application? Better get a comfy chair, because your butt is going to get pretty sore by the time you figure out how, even assuming you can.

Let me give you a concrete example to back up what I’m saying here.

For many years, I used a text-mode e-mail program called mutt, that runs in a terminal window. The first time I used mutt, I couldn’t even figure out how to select which message I wanted to look at, and though it didn’t take too long to learn my way around, it was definitely a project. So, score 1 for Mail.app, which had basically no learning curve at all for selecting and viewing messages.

Now, as it happens, I like to have a signature at the end of my messages. The signature is just a little chunk of text that identifies who I am, and how to reach me. This is such a common desire that most mail programs give you a way to automatically stick a signature at the end of each message you write. In Mutt, you do this by putting the signature into a file, say signature.txt, and adding a line to its configuration file that says, e.g.,

set signature = "signature.txt"

No muss, no fuss. Doing the same thing in Apple’s Mail application requires that you navigate through a complex Preferences dialog box in which, once you have created a new signature text and assigned it a name, you have to choose which mail accounts you will associate it with, and choose what font to display it in, should you decide against all that is good and just to send styled e-mail. Furthermore, if you want your signature to come up automatically on all your messages, you have to manually select each account listed in your preferences and choose that signature as your default signature for new messages. Score 1 for Mutt, for making the simple case easy.

Now comes the interesting part. I want my signature to contain not just the static text identifying me, but also a randomly-chosen quotation. Ideally, I would like it to choose a new quotation for each message I compose. In Mutt, this is easy: I wrote a little program that would pick a quotation at random from a file, and combine it with my static signature. But even if you can’t program at all, there are dozens of such programs available on the web. Making it work with Mutt takes one line of configuration:

set signature = "rsig|"

The “|” at the end tells Mutt that rsig is not a text file, but a program that should be run. The output of that program is used as the signature for your message. How do you accomplish this with Mail? Well…therein lies a story.

The simple fact is, there is no easy way to make Mail.app run a program and use its output as a signature. In theory, you could use AppleScript to change the contents of your signature each time you hit some key, but in practise the AppleScript support in Mail.app is kind of broken, and doesn’t permit you to modify signatures, only read them. I wasted a few hours trolling the web for a good way to get around this, and eventually gave it up as a bad job. But wait! Random signatures are built right in to the program. All you have to do is add a whole pile of signatures one at a time, by hand, in the Preferences dialog, manually drag them to each of your e-mail accounts, and manually set each account to choose its signature at random. Okay, so it doesn’t work the same way as in Mutt, but some allowances have to be made for different designs, right?

Well, no.

See, my quotations file has over 250 entries in it, and I add and remove entries all the time. This is really simple to do, since the file is just plain, unformatted text—I just pop it open in TextEdit and make my changes.* And, because Mutt generates a new signature each time it needs one, changes to the quotations file are immediately available for the next message I compose. To add a quotation in Mail.app, I have to type it in and add it to each account by hand. Worse yet, if I want to delete one, I have to remember what it’s called, and carefully delete it from all my accounts—again, by hand. But the crowning failure is this: If you want to change the static text of your signature, you are forced to manually edit each and every one of the signature entries in each and every one of your mail accounts, using the preferences dialog in Mail.app. That, my friends, is a pain in the posterior.

At this point, if you’re not a programmer, you’ll probably just give up and accept that you can’t have random signature quotations; in essence, permitting the software control you. Or maybe you’ll compromise, and have only a handful of them, so that the editing task is easier. Or you won’t bother changing your quotations when you get tired of them. It’s backward and stupid that you should have to make such concessions—the whole point of the graphical user interface was to make it easier for non-programmers to control the software—but there it is, controlling you. Now if you are a programmer, you should be saying to yourself, “okay, so how do I automate this stupid, repetitive task?”

Naturally, it stands to reason that Mail.app must store these signatures on disk somewhere. There must be a file containing them. You surmise that you could just write a program that will generate all the possible signatures from your quotations file, and replace that file. That wouldn’t be as elegant as generating the signatures on the fly, but at least it means when you make a change, you only have to run your generator program again to fix everything up. And in fact, there is such a file.

Except…

It’s not just one file, it’s a whole directory full of files. It lives in

~/Library/Mail/Signatures

and it contains one distinct file for each signature, plus an index file called SignaturesByAccount.plist that associates signatures with accounts.** But it couldn’t be simple: Both the index file and all of the signature files are stored in XML format, so that you can’t just go in and tweak things with a couple of regular expressions; you’ll have to write some actual code to unpack and repack the property list. Each signature is stored in a so-called “webarchive” file, in which the signature itself is stored as a data-field encoded with Base64, amid a bluster of verbose metadata that might be quite useful if these were, in fact, web archives, but which here serve only to expand a 200-character signature text by a factor of three or four. Furthermore, if you store only the raw text of your signature, and record that it’s a plain-text signature, Mail.app will “helpfully” reformat it for you by throwing away all the line breaks and collapsing spaces and tabs to make a single jumbled paragraph of it. So if you want your signature to keep its linebreaks, you have to convert the signature to HTML before encoding it as Base64 and packing it into the webarchive.

I wrote the program, but this is madness. All of this just for a simple piece of text at the end of your message.

And even after all that, you don’t get quite what you wanted—if you add a new account, you still have to go back and manually drag all the signatures over to it in the Preferences dialog. And every time you add or delete a quotation, or change your base signature text, you have to re-run the program that generates all those idiotic XML files. Oh, sure, you got what you wanted, but what a dance you had to do in order to achieve it! Score 1 for Mutt, and −5 to the developers at Apple for making their application so uncooperative with other software. Not that the Windows equivalent is any better: This kind of thing is endemic to GUI applications in general.

The designers of application software would do well to heed the words of Albert Einstein:

“Make everything as simple as possible, but not simpler.”

You can protect your users from a great deal of the complexity of using the machine, but beyond a certain point, the only way you can protect them is to keep them from using it at all. There are just some things your users will have to learn. On the other hand, you can’t use that as an excuse to slack off—most text-mode software is obtuse not because of any intrinsic complexity in the problems it solves, but because the developers were lazy.*** Writing a good, usable GUI is hard work, and not for the faint of heart. Still, a good GUI does not excuse a program that refuses to play nicely with other programs. After all, those other programs work for the human, too.

In summary, I think there are three important lessons here: First, that you can’t hide all the complexity of software from its users without making it unusable; second, that you shouldn’t punt all the hard work onto the user; and third, that I have far too much free time on my hands. Not only did I waste several hours making random signatures work in Mail.app, but I am now sucking out some of your productivity by writing this. But hey, at least I got it out of my system.

And my signatures work again. But it couldn’t be simple.

^* Actually, that’s a slight lie; I actually edit them in Aquamacs Emacs, but the principle is the same. It’s a program for editing plain, unformatted text files.

^** In a fit of perversity, the setting for whether a signature is “plain text” or “rich text” is kept in the index file, not in the individual signature files. I am not sure I really want to know what manner of drug induced brain failure led to this particular design decision, but I admit I have a certain morbid curiosity.

^*** In the interest of fairness, I want to acknowledge that there are also some very well-designed text-mode user interfaces out there in the world. It is just as hard to design a good text-mode UI as it is to design a good GUI, and possibly harder, since the interaction with the user is more limited in a text display.

Filed by Michael at 22:05 under Diatribe, Personal, Technology
4 Comments