In January, Gabe Weatherhead posted a great way to proofread using BBEdit. His idea is to define a custom “language” with a list of commonly confused words as the “keywords”. So just like how in Python mode BBEdit changes the color of if and else, in Grammar mode it can highlight their and there, or whatever other words you specify.

To do this, you create a plist file and save it into the Language Modules folder inside the BBEdit Application Support folder. Gabe links to his plist file in the post referenced above.

Highlight regular expression matches

You can take this idea one step further. A language module can use a regular expression to define what a “string” should look like. This gives you a lot of flexibility. For example, you can use this to highlight all duplicate words in the document.

Here is the regular expression that matches duplicate words. You put this into the language module plist as the String Pattern.

\b(?P<dupe>\w+)\s+(?P=dupe)\b

The \b is a word anchor, and matches at the beginning or end of a word. Then (?P<dupe>\w+) finds one or more word characters and captures them to the dupe register. Next, \s+ looks for one or more whitespace characters. Finally, (?P=dupe) looks for the contents of the dupe register. The final \b ensures the end of the match is at the end of a word.

BBEdit lets you customize the color scheme for each different language. So even though my strings are usually green, I made them red for the Grammar language.

Here is the language module I use: Grammar.plist.

Google Reader fetches my RSS feed about once every hour. So if I publish a new post, it will be 30 minutes, on average, before the post appears there. If I notice a typo but Google Reader already cached the feed, then I have to wait patiently until the Feed Fetcher returns. In the mean time, everyone reads and makes fun of my mistake.

Here is how to let Google Reader know that it should cache your feed now. This uses a protocol called PubSubHubbub, sometimes abbreviated PSH or PuSH.

It seems like almost no one uses PSH. Even Google, who created the protocol, apparently has forgotten about it. But Google Reader supports it, and almost everyone uses Google Reader, so that makes this useful.

1. Link your feed to a hub

Include a rel="hub" link in your feed to specify a PSH hub. Google runs one at pubsubhubbub.appspot.com. In theory, there could be other hubs, but I don’t know of any.

If you have an Atom feed, put the following line after the feed element open tag:

<link href="http://pubsubhubbub.appspot.com/" rel="hub" />

If you have an RSS feed, make sure that you first specify the Atom namespace in your rss tag:

<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">

Then include the following line after the channel element [open tag]:

<atom:link href="http://pubsubhubbub.appspot.com/" rel="hub" />

Now wait until the Google Bot reads your feed for the changes to take effect. Google Reader sees the link and asks the hub to notify it when the feed changes.

2. Ping the hub

Each time you publish a new post or edit a recent post, run this command at the terminal:

curl -d "hub.mode=publish&hub.url={feed_url}" "http://pubsubhubbub.appspot.com"

Of course, change {feed_url} to the URL for your feed. This command tells the hub to fetch your updated feed. The hub informs Google Reader and any other service that has asked to be notified. In my tests, it took about five seconds for the new (or updated) story to appear in Google Reader.

More information

At https://pubsubhubbub.appspot.com/topic-details?hub.url={feed_url}, you can see the last time the hub checked your feed.

Using PSH makes it so Google Reader polls your site less often, since it assumes you will notify them when something changes. So either go all in or don’t bother.

I use Jumpcut to save and use my clipboard history. One downside to Jumpcut is that it doesn’t play nicely with TextExpander. Because TextExpander manipulates the clipboard, Jumpcut captures every expanded snippet into its history. Some people may be able to tolerate this kind of pollution, but I cannot.

So I don’t use TextExpander. I have often wanted to, but not enough to give up Jumpcut. There are clipboard managers built into Launchbar and Alfred which purportedly play nicely with TextExpander (I can verify Launchbar but not Alfred), but even this cannot persuade me to give up Jumpcut.

Jumpcut is open source, but I know nothing of Objective-C or Mac development. I do know that TextExpander identifies things it puts on the clipboard with org.nspasteboard.AutoGeneratedType. Luckily for me, Wolf Rentzsch, whose podcast I just discovered, made a patch to ignore passwords put on the clipboard by PasswordWallet, which apparently uses a similar identification scheme. It seemed straightforward to adjust his patch to avoid capturing TextExpander snippets.

Surprisingly, it worked. TextExpander, welcome to the team.

Here is the single line I changed in Jumpcut’s AppController.m:

        pbCount = [[NSNumber numberWithInt:[jcPasteboard changeCount]] retain];
        if ( type != nil ) {
        NSString *contents = [jcPasteboard stringForType:type];
-       if ( contents == nil ) {
+       if ( contents == nil || [jcPasteboard stringForType:@"org.nspasteboard.AutoGeneratedType"] ) {
    //            NSLog(@"Contents: Empty");
            } else {

I have a Jumpcut fork on Github. In addition to this change, I have incorporated some stylistic changes from Dr. Drang’s fork.

Obviously you’ll need Xcode to build it. I’d be happy to send a link to a binary if someone wants it, but I don’t really know what I’m doing. All I know is it works for me.

This week, my method for keeping track of journal articles I use went from kind of cool to super awesome thanks to pdftotext and a tiny Perl script.

Most math papers I read come from the arXiv (pronounced archive), a scientific paper repository where the vast majority of mathematicians post their journal articles. This is the best place to find recent papers, since they are often posted here a year or more before they appear in a journal. It is a convenient place to find slightly older papers because journals have terrible websites. (Papers from the 60’s, which I do sometimes read, usually require a trip to the library.)

I use BibDesk to organize the papers I read, mostly because it works so well with Latex, which all mathematicians use to write papers. Also, it stores its database in a plain text file, and has done so since long before it was cool.

Every now and then I gather all the papers from my Dropbox and iPad and import them into BibDesk. For each PDF I got from the arXiv I do the following:

1. Find the arXiv identification number, which is watermarked on the first page of the PDF.

2. Use my script arxiv2bib, which I have written about before to get the paper’s metadata from the arXiv API. An AppleScript takes the result of the script and imports it into BibDesk.

3. Drag the PDF onto the reference in BibDesk. BibDesk automatically renames the paper based on the metadata and moves it to a Dropbox subfolder.

Three steps is better than the ten it would take without AppleScript and the arXiv API, but why can’t the computer extract the identification number automatically?

Oh yeah, of course it can.

#!/bin/bash
pdftotext "$1" - | perl -ne 'if (/^arXiv:(\d{4}\.\d{4}v\d+)/) {print "$1\n"; last}'

The pdftotext utility comes with xpdf, which is available from Homebrew. Or can download the binary linked at foolabs. It works as advertised.

The -n argument tells Perl to wrap the script in the while loop to process stdin one line at a time. Here is what the Perl script would look like if I had put it in its own file.

#!/usr/bin/perl
while (<>) {
    if (/^arXiv:(\d{4}\.\d{4}v\d+)/) {
        print "$1\n";
        last;
    }
}

The regular expression looks for a line beginning with an arXiv identifier, which looks like arXiv:1203.1029v1. If it finds something, it prints the captured part, that is, the actual number. Then it exits the loop.

I can pipe the output of this script into arxiv2bib to fetch the metadata from the arXiv API. An AppleScript glues it all together, allowing me to select a whole bunch of PDFs and run the script. A few seconds later, and all the paper metadata is in BibDesk and the files are renamed and in the proper place.

This week my Mac refused to open a plain text file, complaining that it was from an unidentified developer.

Mac OS X 10.8 introduces a new feature called Gatekeeper, which prevents applications from running unless they are either from the App Store or signed by a registered developer.

You can turn Gatekeeper off, but I have kept it on so far. I am willing to run unsigned Apps, but it is nice to be notified that they are unsigned before having to make that choice.

Don’t open that file!

I never expected to be prohibited from opening a text file. Double-clicking the file got me nowhere. Dragging the file onto BBEdit in my dock did nothing. TextEdit, no. Safari, no. Eventually I right clicked the file and clicked “Open”, which is the prescribed way to get around Gatekeeper’s restriction. Of course this worked, but it opened the file in Byword, which is not my default text editor. I was perplexed.

For this to happen, I found that two things were necessary. One, of course, the file had to be downloaded from the internet. Your web browser sets the “com.apple.quarantine” extended attribute, which tells the computer to be extra careful with this file of dubious origin. Two, the file must be set to open in some application other than the default application. You can change which application is set to open a file by selecting “Get Info” in the Finder, and changing which application appears under “Open with”. This information is stored in the file’s resource fork.

This is clearly a bug. Maybe the operating system would want to prevent someone from tricking me into open a file in something other than my default application, but it should definitely allow me to open it by dragging it onto an application of my choice.

Here is a file that should make this happen on your computer: Harmless File.txt. In this case, I set the file to be opened with Safari, instead of whatever your default text editor is.

If you are curious, you can see the extended attributes by using the command xattr -l "Harmless File.txt" on the command line.

Don’t view that shell script!

There is one other situation that causes similar results. If you have an executable file with no extension, OS X identifies it as a “Unix Executable File” and by default uses Terminal to open it. Gatekeeper also prevents you from opening these files. This makes a little more sense, because opening them in the Terminal actually runs them, which is not what you should do with a random script downloaded from the internet.

What you should do instead is drag them onto a text editor and look at them. But Gatekeeper won’t let you do this either. Worse, if you try to get around Gatekeeper by right clicking and selecting “Open”, the file gets executed in a Terminal window. Oops.

Unquarantine

These seem like edge cases, but they both hit me in the last week, so I created a service in Automator to clear the quarantine bit. (If you download the one I created, you will have to sidestep Gatekeeper, but for valid reasons.)

Here is the shell script from the Automator service:

for f in "$@"
do
    xattr -d com.apple.quarantine "$f"
done

This BBEdit filter allows you to cycle through choices for a script’s shebang line. It is a fun little script, and has been surprisingly useful to me.

BBEdit text filters are shell scripts that read from stdin and write to stdout. When you select a filter from the “Apply Text Filter” submenu of the “Text” menu, the text of the current document is fed through your shell script, and the document text is replaced by the output. If some text is selected, only that part passes through the script.

My Python script is super simple. It reads the first line of the file and determines the name of the interpreter. Then it calculates three possible shebang lines: the stock system command (e.g. /usr/bin/python), the one on my custom path (e.g. /usr/local/bin/python), and the env one (e.g. /usr/bin/env python). Then it cycles from one to the next each time you run it through the filter.

As an added bonus, the script is forgiving. So if I want a Python script, I can just write python on the first line, and the filter will change it to #!/usr/bin/python. This is good for me, because for some reason it always takes me three or four seconds to remember if it should be #! or !#. (At least only one of these makes sense. I have even worse problems remembering the diference between $! and !$ in bash.)

The python script

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

#!/usr/bin/python

import sys
import os

LOCAL = "/usr/local/bin:/usr/local/python/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/texbin:/Users/grigg/bin"
SYSTEM = "/usr/bin:/bin:/usr/sbin:/sbin"

def which(command, path):
    """Emulate the 'which' utility"""
    for p in path.split(':'):
        full_command = os.path.join(p, command)
        if os.path.isfile(full_command) and os.access(full_command, os.X_OK):
            return full_command
    return ""

transformations = [lambda s: which(s, SYSTEM),
                    lambda s: which(s, LOCAL),
                    lambda s: "/usr/bin/env " + s]

# deal with the first line
line = original_line = next(sys.stdin).strip('\n')
if line[:2] == "#!":
    line = line[2:].strip()
base = line.rsplit('/', 1)[-1]
if base[:4] == 'env ':
    base = base[4:].strip()
if ' ' in base:
    base, args = base.split(' ', 1)
    args = ' ' + args
else:
    args = ''

# do the transformations
options = [T(base) for T in transformations]
# filter out the empty ones while appending args
options = [o + args for o in options if o]
# if the only one is the /usr/bin/env, don't do anything
if len(options) <= 1:
    print original_line
else:
    dedupe = list(set(options))
    if line in dedupe:
        dedupe.sort()
        index = dedupe.index(line)
        line = dedupe[(index + 1) % len(dedupe)] # cycle
    else:
        # can't cycle, just use the first option
        line = options[0]
    print "#!" + line

# print every other line
for line in sys.stdin: print line,

The possible transformations are listed beginning on line 17. The order of this list sometimes matters; it determines which transformation should be used if the current shebang doesn’t match any of the options (see line 49).

The current shebang is interpreted on lines 22 through 32. It’s pretty basic, just using the first word after the last slash as the interpreter. That should cover most use cases.

(I am very happy that when I write base[:4] in line 26, Python doesn’t complain if base has fewer than four characters. Contrast this with AppleScript, which fails if base is short and you say text 1 thru 4 of base. You can get around this by testing base begins with "env ". Sorry for the tangent.)

In lines 42 through 46, we get to the cycling. I deduplicate the list, alphabetize, look up the current shebang in the list, and cycle to the next. It is fun how the filter uses the first line of the input file to essentially save state, so it knows where to go next time it is run.

Mountain Lion deleted all my Python packages, just like Lion did last year. I’m determined not to be fooled a third time, so this time I installed my packages to a custom directory using virtualenv.

Virtualenv is advertised as a way to create multiple isolated Python environments on the same computer, switch between them easily, etc. I don’t need all of that. I just want to control where Python packages are installed with no fuss. Virtualenv does that also.

Set up a virtual environment

First, install virtualenv. I recommend installing pip and running pip install virtualenv. If you think it is already installed but it isn’t working, this is probably because Mountain Lion deleted it with the rest of your packages. Reinstall virtualenv.

Second, run virtualenv /usr/local/python. This creates a new Python environment based at /usr/local/python. Of course, you can make this any path you want.

You now have two new directories (among others):

/usr/local/python/bin
/usr/local/python/lib/python2.7/site-packages

The site-packages directory is where your packages will be installed. The bin directory contains a special pip executable, which automatically installs new packages to your custom directory. It also contains a python executable, which will make these packages available to a Python session. Also, if you install any package that has a command line interface, the executable file will go in this bin directory. I added /usr/local/python/bin to (the front of) my PATH to make these easily accessible.

Make these packages available to the system’s Python

Third, create a file local.pth (name doesn’t matter, but extension does) inside the /Library/Python/2.7/site-packages folder with a single line:

/usr/local/python/lib/python2.7/site-packages

This tells the regular system Python to also load packages from my custom location. So even if I run /usr/bin/python, I will be able to import my packages.

As long as I always use the pip command that virtualenv created to install new packages, this third step is the only thing I will have to repeat next year, when OS X 10.9 “Volcano Lion” clears out the system site-packages folder again.

A while back, I wrote about ending lines at natural breaks, like at the end of phrases and sentences. In Latex documents, code documentation, git commits, and other situations where I am writing prose but “soft wrapping” is undesirable, this makes revising and rearranging the text much easier. My favorite part is that I no longer have to worry about refilling the lines every time I make a change.

In that post, I linked to a “new line after punctuation” BBEdit AppleScript I wrote. Some time later, I realized that using AppleScript to run BBEdit searches can be way more efficient than pure AppleScript or even Python alternatives. I thought I would use this to extend my new line script to also handle some math-breaking heuristics. That failed because it got too complicated for me to predict what it was going to do. The most important thing about text editor commands is that they be completely predictable so they don’t interrupt your editing flow.

Still, using BBEdit’s find command simplified my new line script and made it one-third as long. I thought it would be another helpful example of how to script BBEdit, so I’m am posting it here. If you are looking for other examples of this kind of script, Oliver Taylor recently posted a collection of helpful cursor movement scripts, most of which use BBEdit’s find function in a similar way.

The new “new line after punctuation”

tell application "BBEdit"
    -- save location of current cursor
    set cursor to selection
    set cursor_char to characterOffset of cursor

    tell document 1
    set search to find "^([ \\t]*)(.*)([.,!?;:])[ \\t]+" ¬
        searching in text 1 ¬
        options {search mode:grep, backwards:true} without selecting match

    if found of search and startLine of found object of search ¬
        is equal to startLine of cursor then
        -- found punctuation, insert return
        set replacement to grep substitution of "\\1\\2\\3\\r\\1"
        set diff to (replacement's length) ¬
        - (length of found object of search)
        set contents of found object of search to replacement
    else
        -- no punctuation, just insert a return here
        set search to find "^[ \\t]*" searching in text 1 ¬
        options {search mode:grep, backwards:true} without selecting match
        if found of search and startLine of found object of search ¬
        is equal to startLine of cursor then
        set indent to contents of found object of search
        else
        set indent to ""
        end if
        set diff to (length of indent) + 1
        set contents of character cursor_char to ¬
        ("\r" & indent & contents of character cursor_char)
    end if

    -- adjust cursor to keep it in the same relative location
    select insertion point before character (cursor_char + diff)
    end tell
end tell

We begin by telling BBEdit to get the selection object. This contains information about the cursor location if no text is selected. For convenience, we save the characterOffset of the selection, which is the character count from the beginning of the document to the cursor location.

The heavy lifting is done by the grep search in line 7. Here it is with undoubled backslashes, which had been escaped for AppleScript.

^([ \t]*)(.*)([.,!?;:])[ \t]+

The ^ anchors the search to the beginning of a line. The ([ \t]*) captures any combination of spaces and tabs to \1. The (.*) captures as many characters as possible to \2. The ([.,!?;;]) captures a single punctuation mark to \3. The final [ \t]+ requires one or more whitespace characters after the punctuation and captures this whitespace to the search result, but not to any particular group. This is so I don’t insert a new line where there wasn’t already whitespace, and also to throw away whatever whitespace was there.

Lines 8 and 9 search backwards from the current cursor position. If this command (and many that follow) weren’t inside a tell document 1 block, I would need text 1 of document 1 instead of text 1. If you allow BBEdit to select the match, you can see selection flashing while the script runs, which I don’t like. Also, it messes up the cursor location, which will change the result of future searches within the script.

Lines 11 and 12 make sure the script found something on the current line. If so, line 14 uses BBEdit’s grep substitution command to form a string with the whitespace, followed by the line up to the punctuation, followed by the punctuation, a return, and then the whitespace repeated. Line 17 does the replacement.

Lines 20 through 31 deal with the case that no punctuation is found. This just inserts a return at the current cursor location, matching the current line’s indentation.

Line 34 moves the cursor to a new location to make up for the text that has been added.

This script is also available as a gist. I have it assigned to the keyboard shortcut Control+Return.

Here is a Python script to export Day One journal entries into any text-based format (such as html, markdown, plain text, or Latex).

Install

pip  install  dayone_export

View on github

nathangrigg/dayone_export

Download

nathangrigg-dayone_export-latest.zip

Documentation

Read the Docs

Note: the Day One Mac app can export to plain text, and in the future the iOS apps will export to PDF. This is for people who need or want extra customization options.

My Day One journal

I have always been a fan of the Day One Mac and iOS Apps. I was keeping an electronic journal before Day One came along (in Latex, if you must know), but I quickly changed my ways. Day One makes things easier and more fun.

I switched to Day One quickly, but not without an abundance of caution. The last thing I want is for my journal to be unreadable 5 years from now. I was reassured to see that Day One stores each journal entry as a plist file. Furthermore, the entry itself is 100% certified organic plain text. The rest of the plist is just metadata.

As time passed, Day One added features. Most recently, they added the ability to include photos, location information, and weather. All this talk about new features scared me, because more features almost always means more complication. In this case, there was a good chance the extra complication would make my data less future-proof. But in the end, there was no need for me to worry. These guys are good.

A Day One entry is still a simple plist file. The entry itself is still plain text. Location and weather are just more metadata. Best of all, photos can be included with but not inserted into an entry. There is no need for any markup within the journal entry saying “this is where the photo goes.” You don’t have to base64-encode or link to an external file or any of the other awful things word processors have done when dealing with images. A photo is just another piece of metadata that says “This photo goes with that journal entry.”

A Day One export tool

I put together a Python script to export my journal entries. It uses a Jinja template to combine my Day One entries into a single file of whatever format I want. A simple template and a few lines of css turned my journal into enough html to fill 80 printed pages:

I wrote it for myself, but I thought others might find it useful, so I have posted it on github.

Reading plist files in Python

This part is easy. Just import plistlib and do

plistlib.readPlist(filename)

Using Jinja templates in Python

This is more complicated, and the main thing I learned by making this script. The following script fills in a Jinja template:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

from jinja2 import Environment, FileSystemLoader
import os

# j contains list of entries, template is the name of the template file,
# and markdown_filter is a function which converts markdown to html
path, base = os.path.split(template)
env = Environment(loader=FileSystemLoader(path), trim_blocks=True)
env.filters['markdown'] = markdown_filter
template = env.get_template(base)
output = template.render(journal=j)

Loading the template file was a little confusing. First, you set up the environment with a FileSystemLoader in line 7, which takes one or more search paths as an argument. Then the get_template command in line 9 searches through all of the paths for the template you are looking for. I could not find a way to just load an arbitrary template file, hence the awkward workaround in line 6.

Writing Jinja templates

If you’ve ever used Liquid or any other templating language, Jinja looks pretty familiar. Here is a basic template to produce an html version of my journal:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

<!DOCTYPE html>
<html>
<head>
    <title>Journal Entries</title>
    <link rel="stylesheet" href="style.css" type="text/css">
    <meta charset="UTF-8" />
</head>
<body>
<h1 class="page-title">Journal Entries</h1>
{% for entry in journal %}
<article class="entry">
    <h1 class="entry-title">{{ entry['Date'].strftime('%A, %b %e, %Y') }}</h1>
    <p class="location time">
        {% if 'Location' in entry %}
        {{ entry.place(ignore="United States") }},
        {% endif %}
        {{ entry['Date'].strftime("%-I:%M %p %Z") }}
    </p>
    <div class="entry-text">
        {% if 'Photo' in entry %}
        <img class="entry-photo" src="{{ entry['Photo']  }}"/>
        {% endif %}
        {{ entry['Text'] | markdown }}
    </div>
</article>
{% endfor %}
</body>
</html>

Control statements go inside {%...%} blocks, like the for loop from line 10 to line 26 which loops over all the entries. Recall that the journal variable is passed to the template from Python (in line 10 of the Python code above).

Variables are inserted into the document using {{...}} blocks, like the date on line 12 and the photo on line 21. Jinja allows a lot of Python into variable blocks (more than Liquid allows Ruby), which means I can call strftime to format the date in the way that I want. You see more of this on line 14 with the in in the if block, and on line 15 which uses the place method of the entry object.

Line 23 shows how to apply a filter to a variable. The text of the variable passed through the markdown filter. This is a custom filter defined in my Python script, but there are also several built-in filters.

Launchd is Apple’s replacement in OS X for several Unix process management utilities, most notably cron. I use it to run several scripts at scheduled times or fixed intervals. Every day my computer is set to download my twitter statuses and check my library card for overdue books. Every morning my computer resets its volume to medium. Every week it backs up the WordPress database of my “family pictures” blog. It syncs my work files between my computer and the university file server.

Almost anything you can do with cron you can do with launchd, but with more power and flexibility. Unlike cron, launchd does not assume that your computer is always running. So if your computer happens to be sleeping at the time a job is scheduled, it will run the job when it wakes up. This is probably the best feature of launchd, because it allows me to run scripts on my iMac while still letting it sleep when I’m not using it.

I have pieced together what I know about using launchd to schedule jobs from tutorials across the internet, trial and error, and the manuals. This is my attempt to gather all my knowledge about this in one place. If there is something you think I should add or fix, let me know.

(This article has been updated from the original version, most notably with information about new software tools.)

Contents

Because this article is longer than usual.

1. Quick start
2. Launchd basics
3. Tools to manage your agents
   1. Launchctl
   2. LaunchControl
   3. Lingon
   4. Lunchy
4. Format of the plist file
5. Permissions
6. Disabled agents
7. Random thoughts

Quick start

The best way to get started is to buy LaunchControl.

LaunchControl doesn’t have to stay running in the background. It just sets up the job and gets out of the way, letting OS X do the rest. And since OS X already uses launchd to run just about everything, from Spotlight to ssh-agent to the bezel notifications that appear when you change the volume, scheduling jobs will not add any overhead.

Launchd basics

Each launchd agent is stored in an xml plist file. The file contains information about what program to run, when to run it, which arguments to use, and other options. Although technically you can make things work no matter where the plist file is saved, it is best to put it in ~/Library/LaunchAgents, because plists in this folder are automatically loaded into launchd when you log in.

Each agent has a label, which must be unique. Apple uses reverse domain syntax com.apple.whatever, but it doesn’t matter. The plist filename can be anything, but you would be crazy to use anything other than the label, e.g. com.apple.whatever.plist. Sometimes agents are referred to by the file, and sometimes by the label.

Warning: At some point you will use one of your plists as a template to create another. You will of course give them different filenames, but you will forget to change the label, which means only one of them will work. Hopefully this warning will then enter your mind and you will solve the problem.

Apple uses the terms load and unload to mean that an agent is in the system, ready to go, and start or stop to talk about running or killing the actual process. So all agents in your LaunchAgents folder are loaded when the computer starts up. Then it pays attention to when they are scheduled and starts them at the appropriate time. If you create a new plist file you need to load it manually. If you change a plist file, you need to unload it and load it again.

Tools to manage your agents

• Launchctl is Apple’s tool, which gives you most control at the expense of complexity.
• LaunchControl is a graphical interface for managing your agents. It gives you you full control and does a good job of taming the complexity.
• Lingon is another graphical interface for managing your agents.
• Lunchy is like launchctl, but slightly more convenient.

Launchctl

Apple provides launchctl to manage your agents. The main commands you need are

launchctl load ~/Library/LaunchAgents/net.nathangrigg.archive-tweets.plist
launchctl unload ~/Library/LaunchAgents/net.nathangrigg.archive-tweets.plist
launchctl start net.nathangrigg.archive-tweets
launchctl stop net.nathangrigg.archive-tweets
launchctl list

Notice that load and unload require the filename, while start and stop require the label. The start command will manually run the job, even if it isn’t the right time. This can be useful for testing. The stop command just kills the process, but is convenient because you don’t need to know the pid. The list command shows all loaded agents, with the pid if they are currently running and the exit code returned the last time they ran.

LaunchControl

I mentioned LaunchControl earlier, and it is the tool I recommend. It can create and edit your plist files, giving you helpful information when you need it. It can start, stop, load, and unload your agents and help you debug them when they fail.

(LaunchControl was released after the first version of this article, and it is good enough that it makes a lot of what is written here unnecessary.)

Lingon

Lingon is a simple tool for creating and managing launchd agents. It does not give you as much control as LaunchControl, but it has been around longer. When you use Lingon to edit an agent, pressing the Save button automatically loads and unloads the job.

Lunchy

One other useful tool is Lunchy by Mike Perham. This is a ruby script that speeds up the loading and unloading of agents by allowing you to refer to a plist by any unique substring of the filename. My only issue is that it uses terminology that conflicts with Apple’s. Lunchy uses start to mean load and stop to mean unload. It mostly compensates by providing very useful commands restart to unload and then load and edit to edit the file in your default editor.

Install it using gem install lunchy and then edit the file using

lunchy edit archive-tweets

Now reload it using

lunchy restart archive-tweets

Format of the plist file

Here is a very basic plist file to run a script every 86,400 seconds.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>net.nathangrigg.archive-tweets</string>
    <key>ProgramArguments</key>
    <array>
        <string>/Users/grigg/bin/archive-tweets.py</string>
    </array>
    <key>StartInterval</key>
    <integer>86400</integer>
</dict>
</plist>

There are several other keys you can insert between lines 4 and 13 to activate other options. If you want to see what is available, read the launchd.plist manual.

Note that I have provided the full path to the script, since globs aren’t expanded by default. If you want to expand globs, you can include an EnableGlobbing key followed by <true/>.

If your script requires arguments, you would supply these in extra string tags after line 9.

If you download a script from the internet or write one yourself, make sure it is executable, or this might not work.

By default, anything written to standard out or standard error ends up in the system log. If you would like it to be somewhere else, you can use the StandardOutPath and StandardErrorPath keys.

The KeepAlive key allows a script to be run multiple times, depending on certain conditions. If you set it to <true/>, then the script will be run over and over again forever. The following snippet will rerun the script if it returns a nonzero exit code. Read the xml as “Keep the process alive as long as it doesn’t successfully exit”.

<key>KeepAlive</key>
<dict>
    <key>SuccessfulExit</key>
    <false/>
</dict>

In most cases, the system will wait 10 seconds between runs of the script to save system resources. You can adjust this with a ThrottleInterval key, which takes an integer argument, and should be outside the KeepAlive dictionary. You can also set the agent to stay alive depending on the (non)existence of an internet connection using NetworkState or of a file using PathState.

In older versions of OS X, there was an OnDemand key which was required. It is now obsolete and has been replaced by KeepAlive, which is optional. Many of the other examples on the internet still have an OnDemand key, but you don’t need it.

Permissions

For security reasons, launchd will not run LaunchAgents whose plist files have the wrong permissions. For example, they must not be writable by anyone other than the owner. Root LaunchAgents stored in /Library/LaunchAgents must be owned by the root user.

Disabled agents

Both launchctl and lunchy allow you to disable an agent using the -w flag with unload/stop. I do not recommend this. An agent that has been disabled will not load when you log in and cannot be loaded using load/start without using the -w flag again. You will probably just be confused later about why an agent is not loading even though it is in the right place. Information about which agents have been disabled in this manner is stored in a separate file. In Lion, this is the file

/var/db/launchd.db/com.apple.launchd.peruser.NNN/overrides.plist

where NNN is your user id number (find it using id -u).

Random thoughts

Your launchd agents are loaded when you log in, but not unloaded when you log out. So the only time your agents aren’t loaded is during the time between a restart and when you log in. If you have multiple users and need something to run no matter who is logged in, you should put it in /Library/LaunchAgents or /Library/LaunchDaemons.

If your computer sleeps often, it will be asleep when jobs should run, which means it will run them right when it wakes up, possibly before it connects to the internet. I have experimented with KeepAlive and NetworkState to get a job to repeat itself until there is a network connection. You could also use SuccessfulExit and write the script so that it only returns a nonzero code to mean “run again in 10 seconds.” Either method would have the script running (and presumably failing) every 10 seconds when you have no internet connection. A better idea would be to just sleep 5 seconds at the beginning of your script. Or you could double the run frequency and hope for the best.

Here is a fun little experiment you can do while walking home today that shows how much you depend on having two eyes.

Level 1

Pull your keys out of your purse or pocket. While walking at a normal pace, toss your keys up about a foot over your head. Tilt your head toward the place where the keys stop going up and start coming down. Keep walking, don’t move your head, leave your arm at waist level and catch the keys.

I’m not particularly coordinated, but when I did this I was surprised to see that I could catch my keys more than 9 times out of 10, even though I couldn’t see the keys for the last few hundredths of a second as they fell.

Level 2

Now try the same thing, but with one eye closed. Your results will vary, but my success rate fell to 1 in 5. Often I didn’t even manage to touch my keys on the way down.

Last week I was feeling tired of dealing with random AppleScript errors and bugs caused by my complete lack of understanding of the language. I ran across Apple’s AppleScript Language Guide (pdf), which gives a good overview of the language and explains why some things happen they way they do.

I am by no means an AppleScript expert, but here are a few useful things I learned from reading the guide. If you ever AppleScript, set aside an hour to read the guide.

POSIX file, file, and alias

There are three ways to specify a file in AppleScript

alias "Macintosh HD:Users:grigg:Desktop:notes.txt"
POSIX file "/Users/grigg/Desktop/notes.txt"
file "Macintosh HD:Users:grigg:Desktop:notes.txt"

An alias can only refer to a file that already exists. Once an alias is defined during the run of a script, it will refer to the same file even if it is moved or renamed.

Both POSIX file and file refer to filesystem locations instead of files. If you want to refer to a file which does not yet exist, you must use one of these. The only difference between the two is that POSIX file requires paths in POSIX notation, and file requires paths in classic Mac notation.

If you need an alias but prefer to specify paths in POSIX form, you can use

POSIX file "/Users/grigg/Desktop" as alias

If you have an alias or file and want the POSIX name, use POSIX path of the object.

Relative paths are not supported (except when run from a shell; see below) and tilde expansion is not supported.

Run AppleScripts as shell scripts

If you want to run an AppleScript from the command line, you can save the AppleScript as plain text with the interpreter listed in the first line:

#! /usr/bin/osascript

Then make it executable using chmod +x, and run it like you would any other shell script. Of course, your system will need to compile such scripts each time they are run, which in my quick experiment added only 10 to 30 milliseconds of overhead.

If you want to store your AppleScript as plain text but run a compiled version, you can use osacompile. If script.applescript is saved as plain text, then

osacompile -o script.scpt script.applescript

saves a compiled version as a binary file script.scpt. To run the compiled version, you will need to run osascript script.scpt.

AppleScripts run from the command line print their return value (or the result of the last expression, if there is no explicit return) to the output stream.

AppleScripts run from the command line can accept arguments (which are passed as a list to the run handler) and can use relative paths. The following script prints the full Mac-style path to the file listed as first argument.

#! /usr/bin/osascript

on run(arguments)
    set filename to POSIX file (first item of arguments) as alias
    return filename as string
end run

Use pipes as variable delimiters

AppleScript allows you to define variables using pipes as delimiters, for example |variable|, |my variable|, or |2^5|. Naming a variable |2^5| is probably not a good idea, but pipes can still be useful because of the way the AppleScript Editor treats them.

AppleScript reserves a lot of words for its own use. If you are sending commands to another application, this will also reserve a bunch of words. So if you need a variable name, there is a good chance you will accidentally use something you aren’t supposed to, which results in cryptic and annoying errors. Some people get around this by prefixing variables with my or the, as in theFile or myInteger. I have been known to use excessive underscores.

Another way around this issue is to use bars. The best thing is that AppleScript Editor removes them if they are not necessary. So you you write |variable| and nobody else is claiming this word, the editor will remove the bars when you compile or run, giving you confirmation that you are free to use this word.

Reference to a variable

AppleScript deals with variables pretty much the same as Python. Lists and records (dictionaries) are mutable and passed by reference. Strings and numbers are immutable and essentially passed by value.

For example,

set a to {1,2}
set b to a
copy 3 to end of a
-- now a and b are both {1,2,3}

You can use copy a to b to create a deep copy or set b to items of a to create a shallow copy.

In AppleScript you can force pass by reference using the term a reference to.

on change_to_5(b)
    set contents of b to 5
end change_to_5

set a to 0
change_to_5(a reference to a)
-- Now a = 5

This can also be useful when dealing with long lists. According to the Language Guide, the most efficient way to append to a list is

copy 5 to end of (a reference to long_list)

I don’t understand why.

Extract elements of a given type from a list

I don’t know if I would ever actually use this, but I thought it was interesting. If you write

set mixed_set to {1, 1.2, 3, "hello", 5}
integers of mixed_set

then AppleScript returns {1, 3, 5}. You can also write integer 3 of mixed_set to get 5.

I have a love-hate relationship with AppleScript. The syntax is annoying, string manipulation is a pain, it is hard to debug, and scripts are stored in a binary format. But I would be incredibly sad if AppleScript went away. Interacting with other programs gives you a lot of power.

A recent example

Following the advice of Bram Moolenaar, I’ve been monitoring tasks that I repeat often which take longer than they could. I noticed that when writing Latex, I often need to change then name of an environment while editing. For example, I might have

\begin{equation*}
    \int_0^x e^{t^2} \, dt
\end{equation*}

Then I decide that I need to change it to an align* environment or just remove the star from the equation* environment, which means I have to edit both the begin tag and the end tag. Changing both at once will probably only save a few seconds, but those few seconds are saved over and over again, and typing becomes a little less repetitive and a little more productive.

Here is where AppleScript comes in. I need to get the location of the cursor, determine which environment contains it, prompt the user for a new name, make the change, and put the cursor back where it belongs.

The first two of these I could do with Python. BBEdit uses environment variables to pass information about the cursor to cursor position to shell scripts. Python would have no problem doing the searching. But prompting the user and making sure the cursor doesn’t jump due to a change in the length of the document require AppleScript.

The “Change environment” script is a new addition to my Latex BBEdit package, which incorporates lots of Latex related scripts and clippings I’ve made and collected. I’ll explain some of the useful details of the script in this post.

Get and set BBEdit’s cursor location

You can use AppleScript to get the number of characters between the beginning of the document and the current cursor location:

tell application "BBEdit"
    set cursor_loc to characterOffset of selection
end tell

You can move the cursor around with the command select insertion point. In my case, if I replace equation* with align*, I would like to shift the cursor left by 3, so that its relative position stays the same.

tell application "BBEdit"
    select insertion point before character (cursor_loc + diff) ¬
        of document 1
end tell

Search BBEdit document using AppleScript

To figure out which environment contains the cursor, I search backwards from the cursor for a begin, then forward from that point for an end, and then I check to make sure that the cursor is between the two. If it is not, I repeat the process until it is. There is a little extra logic in there to deal with possible nested environments.

A basic search looks like

tell application "BBEdit"
    find "\\\\begin{equation*}" searching in ¬
        characters 1 thru cursor_loc of document 1
end tell

The result is an AppleScript “record” (dictionary) with keys found (was there a match?), found object, and found text. The found object has properties like characterOffset and length and startLine.

Notice the double escaped backslashes. AppleScript interprets each pair of backslashes as a single backslash, and BBEdit does the same. This sort of thing used to drive me nuts, but I’m better at it now.

Of course, things don’t really get exciting until you do a grep search. This one searches backwards from the cursor for any sort of begin and captures the name of the environment.

tell application "BBEdit"
    set match to find "\\\\begin{(.*?)}" searching in ¬
        characters 1 thru cursor_location of document 1 ¬
        options {search mode:grep, backwards: true}

    if found of match then
        set environment to grep substitution of "\\1"
    end if
end tell

Change the text

Here is how you change just one piece of the document

tell application "BBEdit"
    set characters 10 thru 15 of document 1 to "new text"
end tell

Calendaring is messy. You have one-time events, repeating events, all-day events, event locations, event attendees, time zones, alarms, and so on. To manage this complexity, you need an equally complex calendar application. For basic use, I get by (I actually use Apple’s iOS and OS X apps), but sometimes I need something more basic.

One situation in which almost every calendar application will make things difficult is making a calendar for a class I’m teaching. It would be difficult to create an event for each class meeting, with the topic we will discuss in the description of each event. It would be even harder when, five classes in, I decide we need to do something different one day and thus shift the topic of every remaining class back by one day.

Another situation is managing birthdays. I have a very large family (more than 70 if you count only siblings, in-laws, and their children) and would like to keep track of their birthdays (and ages if possible). At first this seems like a good use of a standard calendar app because you only have to enter something when someone new joins the family. Ages are harder, but you can just put the birth year in the event and do some subtraction. The problem is that I sometimes forget to add someone and it is impossible to notice until next year. Before I know it, I only have the birthdays of half of my nephews under 5, and I have no way of knowing (besides exhaustive search) which ones I am missing.

What I need is some kind of markdown for calendars. A plain text way to manage dates. Not for all my events, because my apps work well under most circumstances. I just need something for edge cases.

Of course, calendars can be exchanged via the iCalendar format, which is plain text (not related to Apple’s iCal Mac app). But this is a complicated format with the ability to do every crazy thing that I imagine CEOs make their secretaries do. It is not meant to be managed by hand.

When

Here is where a Perl script named when comes in. From the project website:

When is an extremely simple personal calendar program, aimed at the Unix geek who wants something minimalistic.

Count me in.

If you are using OS X and homebrew, you can install it with brew install when. It has no dependencies, so you could almost as easily install it yourself. (By install I mean “copy to your computer”, assuming you have Perl. Speaking of which, how is it possible that Windows doesn’t ship with Perl, Python, and Ruby? Man, I could never go back.)

When reads events stored in a plain text file. Each event is a single line, and repeating events are defined using either cron-like syntax (2012 * 01) or equations (y=2012 & d=1). It is efficient, Unix-y and beautiful. If you stick to the cron-like syntax, sorting the lines alphabetically sorts events chronologically, to the extent that repeating events can be sorted.

There is a special syntax for annual repeats that allows you to reference the year. So

1932* 06 10, Pierre Cartier (\a)

creates a repeating event, and the when utility calculates the age like this:

$ when
Thu 2012 Jun 7 6:31

Sun 2012 Jun 10 Pierre Cartier (80)

This is a perfect solution to my birthday problem. Family members can be sorted by age, which makes it easy to tell who needs added. It would be simple to convert the file to a basic iCalendar file which I could subscribe to in iCal or my parents could subscribe to in Google.

This would make a class calendar much easier also.

I recently learned about BBEdit’s menu scripts, which allow you to run an AppleScript before or after you do anything that uses one of BBEdit’s menus.

I write a lot of things in Latex using BBEdit, and many of these need to be printed. Since I am using BBEdit, my impulse is to print the document from the BBEdit menu (or by pressing Command+P while BBEdit is active). When I get around to collecting the pages from the printer, I find a beautiful color printout of the Latex source.

I have tried to retrain myself and failed, so it is time for an intervention.

BBEdit menu scripts

To make a menu script, create an AppleScript named after the menu item to which you want to attach it. In my case, I name it File•Print….scpt. Notice the use of the bullet (Option+8) to separate the menu from the menu item, and the ellipsis (Option+;) to match the menu name exactly. The script should live in the “Menu Items” folder in your BBEdit Application Support folder, which you may have to create.

The script should have a menuSelect handler. If the handler returns True, then the script replaces the menu item, that is, the menu item is not run. If the handler returns false, the the menu item is run after finishing the script.

My script checks if I am trying to print a Tex document, and if so, shows an “Are you sure” dialog. If I really do want to print, the script returns False, and the print process continues.

The script is not complicated. It checks to make sure the document is a Tex document, and if so, displays a dialog. It returns True or False based on the user’s input.

on menuSelect()
    set skip_print to false
    tell application "BBEdit" to get source language of document 1
    if result is equal to "TeX" then
        try
            display dialog ¬
                "Are you sure you want to print the TeX source?" ¬
                with title "BBEdit Print" ¬
                buttons {"Don't Print", "Print"} ¬
                default button 2 ¬
                cancel button 1 ¬
                with icon caution
            get button returned of result
            set skip_print to (result is not "Print")
        on error --if they cancel the dialog
            set skip_print to true
        end try
    end if
    return skip_print
end menuSelect

You can use postmenuSelect to specify something to be run after BBEdit completes the menu item.

The moral of the story

For those of us who spend a lot of their computer time writing and manipulating text, having a good text editor and tweaking it to meet our needs can save a lot of time and frustration.

It is also very satisfying. I put in a little bit of time up front to save myself time repeatedly in the future. Then every time I use one of these little solutions, I imagine my past self saying to my current self, “Hey, let me do that for you.” This turns what would have been a tiny frustration into a tiny success.