I use Fastmail for my personal email, and I like to keep a backup of my email on my personal computer. Why make a backup? When I am done reading or replying to an email, I make a split-second decision on whether to delete or archive it on Fastmail’s server. If it turns out I deleted something that I need later, I can always look in my backup. The backup also predates my use of Fastmail and serves as a service-independent store of my email.

My old method of backing up the email was to forward all my email to a Gmail account, then use POP to download the email with a hacked-together script. This had the added benefit that the Gmail account also served as a searchable backup.

Unfortunately the Gmail account ran out of storage and the POP script kept hanging for some reason, which together motivated me to get away from this convoluted backup strategy.

The replacement script uses JMAP to connect directly to Fastmail and download all messages. It is intended to run periodically, and what it does is pick an end time 24 hours in the past, download all email older than that, and then record the end time. The next time it runs, it searches for mail between the previous end time and a new end time, which is again 24 hours in the past.

Why pick a time in the past? Well, I’m not confident that if you search up until this exact moment, you are guaranteed to get every message. A message could come in, then two seconds later you send a query, but it hits a server that doesn’t know about your message yet. I’m sure an hour is more than enough leeway, but since this is a backup, we might as well make it a 24-hour delay.

Note that I am querying all mail, regardless of which mailbox it is in, so even if I have put a message in the trash, my backup script will find it and download it.

JMAP is a modern JSON-based replacement for IMAP and much easier to use, such that the entire script is 135 lines, even with my not-exactly-terse use of Python.

Here is the script, with some notes below.

  1 import argparse
  2 import collections
  3 import datetime
  4 import os
  5 import requests
  6 import string
  7 import sys
  8 import yaml
  9 
 10 
 11 Session = collections.namedtuple('Session', 'auth account_id download_template')
 12 def get_session(username, password):
 13     auth = requests.auth.HTTPBasicAuth(username, password)
 14     r = requests.get('https://jmap.fastmail.com/.well-known/jmap', auth=auth)
 15     [account_id] = list(r.json()['accounts'])
 16     download_template = r.json()['downloadUrl']
 17     return Session(auth, account_id, download_template)
 18 
 19 
 20 Email = collections.namedtuple('Email', 'id blob_id date subject')
 21 def query(session, start, end):
 22     json_request = {
 23         'using': ['urn:ietf:params:jmap:core', 'urn:ietf:params:jmap:mail'],
 24         'methodCalls': [
 25             [
 26                 'Email/query',
 27                 {
 28                     'accountId': session.account_id,
 29                     'sort': [{'property': 'receivedAt', 'isAscending': False}],
 30                     'filter': {
 31                         'after': start.isoformat() + 'Z',
 32                         'before': end.isoformat() + 'Z',
 33                     },
 34                     'limit': 50,
 35                 },
 36                 '0',
 37             ],
 38             [
 39                 'Email/get',
 40                 {
 41                     'accountId': session.account_id,
 42                     '#ids': {
 43                         'name': 'Email/query',
 44                         'path': '/ids/*',
 45                         'resultOf': '0',
 46                     },
 47                     'properties': ['blobId', 'receivedAt', 'subject'],
 48                 },
 49                 '1',
 50             ],
 51         ],
 52     }
 53 
 54     while True:
 55         full_response = requests.post(
 56             'https://jmap.fastmail.com/api/', json=json_request, auth=session.auth
 57         ).json()
 58 
 59         if any(x[0].lower() == 'error' for x in full_response['methodResponses']):
 60             sys.exit(f'Error received from server: {full_response!r}')
 61 
 62         response = [x[1] for x in full_response['methodResponses']]
 63 
 64         if not response[0]['ids']:
 65             return
 66 
 67         for item in response[1]['list']:
 68             date = datetime.datetime.fromisoformat(item['receivedAt'].rstrip('Z'))
 69             yield Email(item['id'], item['blobId'], date, item['subject'])
 70 
 71         # Set anchor to get the next set of emails.
 72         query_request = json_request['methodCalls'][0][1]
 73         query_request['anchor'] = response[0]['ids'][-1]
 74         query_request['anchorOffset'] = 1
 75 
 76 
 77 def email_filename(email):
 78     subject = email.subject.translate(str.maketrans('', '', string.punctuation))[:50]
 79     date = email.date.strftime('%Y%m%d_%H%M%S')
 80     return f'{date}_{email.id}_{subject}.eml'
 81 
 82 
 83 def download_email(session, email, folder):
 84     r = requests.get(
 85         session.download_template.format(
 86             accountId=session.account_id,
 87             blobId=email.blob_id,
 88             name='email',
 89             type='application/octet-stream',
 90         ),
 91         auth=session.auth,
 92     )
 93 
 94     with open(os.path.join(folder, email_filename(email)), 'wb') as fh:
 95         fh.write(r.content)
 96 
 97 
 98 if __name__ == '__main__':
 99     # Parse args.
100     parser = argparse.ArgumentParser(description='Backup jmap mail')
101     parser.add_argument('--config', help='Path to config file', nargs=1)
102     args = parser.parse_args()
103 
104     # Read config.
105     with open(args.config[0], 'r') as fh:
106         config = yaml.safe_load(fh)
107 
108     # Compute window.
109     session = get_session(config['username'], config['password'])
110     delay_hours = config.get('delay_hours', 24)
111 
112     end_window = datetime.datetime.utcnow().replace(microsecond=0) - datetime.timedelta(
113         hours=delay_hours
114     )
115 
116     # On first run, 'last_end_time' wont exist; download the most recent week.
117     start_window = config.get('last_end_time', end_window - datetime.timedelta(weeks=1))
118 
119     folder = config['folder']
120 
121     # Do backup.
122     num_results = 0
123     for email in query(session, start_window, end_window):
124         # We want our search window to be exclusive of the right endpoint.
125         # It should be this way in the server, according to the spec, but
126         # Fastmail's query implementation is inclusive of both endpoints.
127         if email.date == end_window:
128             continue
129         download_email(session, email, folder)
130         num_results += 1
131     print(f'Archived {num_results} emails')
132 
133     # Write config
134     config['last_end_time'] = end_window
135     with open(args.config[0], 'w') as fh:
136         yaml.dump(config, fh)

The get_session function is run once at the beginning of the script, and fetches some important data from the server including the account ID and a URL to use to download individual emails.

The query function does the bulk of the work, sending a single JSON request multiple times to page through the search results. It is actually a two-part request, first Email/query, which returns a list of ids, and then Email/get, which gets some email metadata for each result. I wrote this as a generator to make the main part of my script simpler. The paging is performed by capturing the ID of the final result of one query, and asking the next query to start at that position plus one (lines 73-74). We are done when the query returns no results (line 64).

The download_email function uses the blob ID to fetch the entire email and saves it to disk. This doesn’t really need to be its own function, but it will help if I later decide to use multiple threads to do the downloading.

Finally, the main part of the script reads configuration from a YAML file, including the last end time. It loops through the results of query, calling download_email on each result. Finally, it writes the configuration data back out to the YAML file, including the updated last_end_time.

I understand the popularity of email newsletters, especially for publishers. It’s a simple way to get paid content out, easier for users than a private RSS feed. But that doesn’t mean I want to read newsletters in my email app.

Feedbin, which I am already using for my regular RSS subscriptions, bridges the gap. As part of my Feedbin account, I get a secret email address, and anything sent to that address ends up in my RSS reader. Problem solved!

But it quickly gets annoying to sign up for newsletters (often creating an account) with an email address that is neither memorable nor truly mine. Fastmail, which I am already using for my regular email, makes it easy to find specified emails sent to my regular address, forward them to my feedbin address, and put the original in the trash.

In fact, Fastmail lets me use “from a member of a given contact group” as the trigger for this automatic rule, which makes the setup for a new newsletter very simple:

1. Subscribe to the newsletter
2. Add the sender to my Fastmail address book
3. Add the newly created contact to my “Feedbin” group

This is very convenient, for newsletters as well as other mail that is more of a notification than an email. Here are some of the emails that I now read as though they were feeds:

• Ben Thompson’s Stratechery
• Craig Calcaterra’s Cup of Coffee (baseball)
• Noah Smith’s Noahpinion
• A handful of Washington Post newsletters
• Los Angeles County Covid updates
• UPS My Choice emails

This is the fourth and last in a series of posts describing the system I built to monitor long-running commands.

The third post explained my mon script, which together with my twait script from the second post, lights up an LED when a command in a tmux pane or iTerm window completes.

If I want to monitor a command, I use mon twait ID, where ID is the pane or window ID. Or I can run another command only if the first command is successful, by writing twait ID && ./other_command.

Of course it would be annoying to type the window ID manually, so I have two keyboard shortcuts. One to start mon twait ID in a separate window that closes automatically when it is done. The other to open a new window with the text twait ID && followed by a space.

Tmux integration

Tmux can be scripted through the use of unix-like commands, with arguments and options. These can either be run from the command line or assigned keyboard shortcuts.

I use Ctrl-A plus uppercase E for the first.

bind E run "tmux split-window -l 5 \"mon twait #D\""

This splits to form a new window, 5 lines high, and runs mon twait. The #D parameter in a tmux run command gets translated into the pane ID.

I use Ctrl-A plus lowercase E for the second.

bind e run "tmux split-window; tmux send-keys \"twait #D &&\""

iTerm integration

You can do very similar things with iTerm using AppleScript.

To open the monitor in a temporary pane, I use this script. It is possible to control the height of the new pane, but I haven’t figured it out yet.

1 tell application "iTerm"
2   tell current session of current window
3     set session_id to id
4     split horizontally with default profile ¬
5         command "mon twait " & session_id
6   end tell
7 end tell

I prefer new windows over tabs or split windows, so I use this script to open a new window prepped to run a follow-up command.

1 tell application "iTerm"
2     set session_id to id of current session of current window
3     set new_window to create window with default profile
4     set new_session to current session of new_window
5     tell new_session
6         select
7         write text "twait " & session_id & " && " without newline
8     end tell
9 end tell

This is the third in a series of posts describing the system I built to monitor long-running commands.

The second post explained my twait script, which waits for a command in a separate tmux pane or iTerm window to complete, and then exits with the same code.

The easiest way to get notified when twait finishes is to chain the commands:

twait %22; notify

But this ignores the return value, which is a useful part of the notification.

You could solve this by passing the return value as an argument:

twait %22; notify $?

The $? variable contains the return code of the last command.

I mostly use a script called mon that takes the other command as an argument:

mon twait %22

Since mon is running the command, it has access to its exit code. It can also do some prep work before running the command. The downside is that you have to keep the command fairly simple; using pipes, multiple commands, or shell aliases does not work properly.

Here is the code to my monitor script, which lights my blink LED and sends a Pushover notification.

 1 #!/bin/bash
 2 set -eu
 3 
 4 function blink() {
 5   blink1-tool -m 0 "$@" > /dev/null
 6 }
 7 
 8 blink --rgb 170,170,170
 9 echo "Monitoring $@"
10 "$@"
11 rc=$?
12 
13 case $rc in
14   0)
15     blink --green
16     title="Process succeeded"
17     ;;
18   1)
19     blink --red
20     title="Process failed"
21     ;;
22   *)
23     blink --blue
24     title="Process failed"
25     ;;
26 esac
27 
28 pushover "$title" "Exited with code $rc"

This sets the LED to a dim white at the beginning of the command, prints a message, and then runs the command. The $@ variable contains all of the arguments to the script, which should be the command to run, followed by its arguments. When we are done with the command, we capture the return code.

Depending on the value of the return code, I turn the LED green, red, or blue, and send a message to my phone and watch.

The pushover script uses curl to send a request to the pushover servers.

1 #!/bin/bash
2 set -eu
3 curl -s \
4   --form-string "token=asdfasdfasdfasdf" \
5   --form-string "user=asdfasdfasdfasdff" \
6   --form-string "title=$1" \
7   --form-string "message=$2" \
8   "https://api.pushover.net/1/messages.json" > /dev/null

Those are not my token or userid. You have to get your own from pushover.

The final post in this series will show some tmux and AppleScript integrations to make this monitoring easier to launch.