Here is a fun little experiment you can do while walking home today that shows how much you depend on having two eyes.

Level 1

Pull your keys out of your purse or pocket. While walking at a normal pace, toss your keys up about a foot over your head. Tilt your head toward the place where the keys stop going up and start coming down. Keep walking, don’t move your head, leave your arm at waist level and catch the keys.

I’m not particularly coordinated, but when I did this I was surprised to see that I could catch my keys more than 9 times out of 10, even though I couldn’t see the keys for the last few hundredths of a second as they fell.

Level 2

Now try the same thing, but with one eye closed. Your results will vary, but my success rate fell to 1 in 5. Often I didn’t even manage to touch my keys on the way down.

Last week I was feeling tired of dealing with random AppleScript errors and bugs caused by my complete lack of understanding of the language. I ran across Apple’s AppleScript Language Guide (pdf), which gives a good overview of the language and explains why some things happen they way they do.

I am by no means an AppleScript expert, but here are a few useful things I learned from reading the guide. If you ever AppleScript, set aside an hour to read the guide.

POSIX file, file, and alias

There are three ways to specify a file in AppleScript

alias "Macintosh HD:Users:grigg:Desktop:notes.txt"
POSIX file "/Users/grigg/Desktop/notes.txt"
file "Macintosh HD:Users:grigg:Desktop:notes.txt"

An alias can only refer to a file that already exists. Once an alias is defined during the run of a script, it will refer to the same file even if it is moved or renamed.

Both POSIX file and file refer to filesystem locations instead of files. If you want to refer to a file which does not yet exist, you must use one of these. The only difference between the two is that POSIX file requires paths in POSIX notation, and file requires paths in classic Mac notation.

If you need an alias but prefer to specify paths in POSIX form, you can use

POSIX file "/Users/grigg/Desktop" as alias

If you have an alias or file and want the POSIX name, use POSIX path of the object.

Relative paths are not supported (except when run from a shell; see below) and tilde expansion is not supported.

Run AppleScripts as shell scripts

If you want to run an AppleScript from the command line, you can save the AppleScript as plain text with the interpreter listed in the first line:

#! /usr/bin/osascript

Then make it executable using chmod +x, and run it like you would any other shell script. Of course, your system will need to compile such scripts each time they are run, which in my quick experiment added only 10 to 30 milliseconds of overhead.

If you want to store your AppleScript as plain text but run a compiled version, you can use osacompile. If script.applescript is saved as plain text, then

osacompile -o script.scpt script.applescript

saves a compiled version as a binary file script.scpt. To run the compiled version, you will need to run osascript script.scpt.

AppleScripts run from the command line print their return value (or the result of the last expression, if there is no explicit return) to the output stream.

AppleScripts run from the command line can accept arguments (which are passed as a list to the run handler) and can use relative paths. The following script prints the full Mac-style path to the file listed as first argument.

#! /usr/bin/osascript

on run(arguments)
    set filename to POSIX file (first item of arguments) as alias
    return filename as string
end run

Use pipes as variable delimiters

AppleScript allows you to define variables using pipes as delimiters, for example |variable|, |my variable|, or |2^5|. Naming a variable |2^5| is probably not a good idea, but pipes can still be useful because of the way the AppleScript Editor treats them.

AppleScript reserves a lot of words for its own use. If you are sending commands to another application, this will also reserve a bunch of words. So if you need a variable name, there is a good chance you will accidentally use something you aren’t supposed to, which results in cryptic and annoying errors. Some people get around this by prefixing variables with my or the, as in theFile or myInteger. I have been known to use excessive underscores.

Another way around this issue is to use bars. The best thing is that AppleScript Editor removes them if they are not necessary. So you you write |variable| and nobody else is claiming this word, the editor will remove the bars when you compile or run, giving you confirmation that you are free to use this word.

Reference to a variable

AppleScript deals with variables pretty much the same as Python. Lists and records (dictionaries) are mutable and passed by reference. Strings and numbers are immutable and essentially passed by value.

For example,

set a to {1,2}
set b to a
copy 3 to end of a
-- now a and b are both {1,2,3}

You can use copy a to b to create a deep copy or set b to items of a to create a shallow copy.

In AppleScript you can force pass by reference using the term a reference to.

on change_to_5(b)
    set contents of b to 5
end change_to_5

set a to 0
change_to_5(a reference to a)
-- Now a = 5

This can also be useful when dealing with long lists. According to the Language Guide, the most efficient way to append to a list is

copy 5 to end of (a reference to long_list)

I don’t understand why.

Extract elements of a given type from a list

I don’t know if I would ever actually use this, but I thought it was interesting. If you write

set mixed_set to {1, 1.2, 3, "hello", 5}
integers of mixed_set

then AppleScript returns {1, 3, 5}. You can also write integer 3 of mixed_set to get 5.

I have a love-hate relationship with AppleScript. The syntax is annoying, string manipulation is a pain, it is hard to debug, and scripts are stored in a binary format. But I would be incredibly sad if AppleScript went away. Interacting with other programs gives you a lot of power.

A recent example

Following the advice of Bram Moolenaar, I’ve been monitoring tasks that I repeat often which take longer than they could. I noticed that when writing Latex, I often need to change then name of an environment while editing. For example, I might have

\begin{equation*}
    \int_0^x e^{t^2} \, dt
\end{equation*}

Then I decide that I need to change it to an align* environment or just remove the star from the equation* environment, which means I have to edit both the begin tag and the end tag. Changing both at once will probably only save a few seconds, but those few seconds are saved over and over again, and typing becomes a little less repetitive and a little more productive.

Here is where AppleScript comes in. I need to get the location of the cursor, determine which environment contains it, prompt the user for a new name, make the change, and put the cursor back where it belongs.

The first two of these I could do with Python. BBEdit uses environment variables to pass information about the cursor to cursor position to shell scripts. Python would have no problem doing the searching. But prompting the user and making sure the cursor doesn’t jump due to a change in the length of the document require AppleScript.

The “Change environment” script is a new addition to my Latex BBEdit package, which incorporates lots of Latex related scripts and clippings I’ve made and collected. I’ll explain some of the useful details of the script in this post.

Get and set BBEdit’s cursor location

You can use AppleScript to get the number of characters between the beginning of the document and the current cursor location:

tell application "BBEdit"
    set cursor_loc to characterOffset of selection
end tell

You can move the cursor around with the command select insertion point. In my case, if I replace equation* with align*, I would like to shift the cursor left by 3, so that its relative position stays the same.

tell application "BBEdit"
    select insertion point before character (cursor_loc + diff) ¬
        of document 1
end tell

Search BBEdit document using AppleScript

To figure out which environment contains the cursor, I search backwards from the cursor for a begin, then forward from that point for an end, and then I check to make sure that the cursor is between the two. If it is not, I repeat the process until it is. There is a little extra logic in there to deal with possible nested environments.

A basic search looks like

tell application "BBEdit"
    find "\\\\begin{equation*}" searching in ¬
        characters 1 thru cursor_loc of document 1
end tell

The result is an AppleScript “record” (dictionary) with keys found (was there a match?), found object, and found text. The found object has properties like characterOffset and length and startLine.

Notice the double escaped backslashes. AppleScript interprets each pair of backslashes as a single backslash, and BBEdit does the same. This sort of thing used to drive me nuts, but I’m better at it now.

Of course, things don’t really get exciting until you do a grep search. This one searches backwards from the cursor for any sort of begin and captures the name of the environment.

tell application "BBEdit"
    set match to find "\\\\begin{(.*?)}" searching in ¬
        characters 1 thru cursor_location of document 1 ¬
        options {search mode:grep, backwards: true}

    if found of match then
        set environment to grep substitution of "\\1"
    end if
end tell

Change the text

Here is how you change just one piece of the document

tell application "BBEdit"
    set characters 10 thru 15 of document 1 to "new text"
end tell

Calendaring is messy. You have one-time events, repeating events, all-day events, event locations, event attendees, time zones, alarms, and so on. To manage this complexity, you need an equally complex calendar application. For basic use, I get by (I actually use Apple’s iOS and OS X apps), but sometimes I need something more basic.

One situation in which almost every calendar application will make things difficult is making a calendar for a class I’m teaching. It would be difficult to create an event for each class meeting, with the topic we will discuss in the description of each event. It would be even harder when, five classes in, I decide we need to do something different one day and thus shift the topic of every remaining class back by one day.

Another situation is managing birthdays. I have a very large family (more than 70 if you count only siblings, in-laws, and their children) and would like to keep track of their birthdays (and ages if possible). At first this seems like a good use of a standard calendar app because you only have to enter something when someone new joins the family. Ages are harder, but you can just put the birth year in the event and do some subtraction. The problem is that I sometimes forget to add someone and it is impossible to notice until next year. Before I know it, I only have the birthdays of half of my nephews under 5, and I have no way of knowing (besides exhaustive search) which ones I am missing.

What I need is some kind of markdown for calendars. A plain text way to manage dates. Not for all my events, because my apps work well under most circumstances. I just need something for edge cases.

Of course, calendars can be exchanged via the iCalendar format, which is plain text (not related to Apple’s iCal Mac app). But this is a complicated format with the ability to do every crazy thing that I imagine CEOs make their secretaries do. It is not meant to be managed by hand.

When

Here is where a Perl script named when comes in. From the project website:

When is an extremely simple personal calendar program, aimed at the Unix geek who wants something minimalistic.

Count me in.

If you are using OS X and homebrew, you can install it with brew install when. It has no dependencies, so you could almost as easily install it yourself. (By install I mean “copy to your computer”, assuming you have Perl. Speaking of which, how is it possible that Windows doesn’t ship with Perl, Python, and Ruby? Man, I could never go back.)

When reads events stored in a plain text file. Each event is a single line, and repeating events are defined using either cron-like syntax (2012 * 01) or equations (y=2012 & d=1). It is efficient, Unix-y and beautiful. If you stick to the cron-like syntax, sorting the lines alphabetically sorts events chronologically, to the extent that repeating events can be sorted.

There is a special syntax for annual repeats that allows you to reference the year. So

1932* 06 10, Pierre Cartier (\a)

creates a repeating event, and the when utility calculates the age like this:

$ when
Thu 2012 Jun 7 6:31

Sun 2012 Jun 10 Pierre Cartier (80)

This is a perfect solution to my birthday problem. Family members can be sorted by age, which makes it easy to tell who needs added. It would be simple to convert the file to a basic iCalendar file which I could subscribe to in iCal or my parents could subscribe to in Google.

This would make a class calendar much easier also.

I recently learned about BBEdit’s menu scripts, which allow you to run an AppleScript before or after you do anything that uses one of BBEdit’s menus.

I write a lot of things in Latex using BBEdit, and many of these need to be printed. Since I am using BBEdit, my impulse is to print the document from the BBEdit menu (or by pressing Command+P while BBEdit is active). When I get around to collecting the pages from the printer, I find a beautiful color printout of the Latex source.

I have tried to retrain myself and failed, so it is time for an intervention.

BBEdit menu scripts

To make a menu script, create an AppleScript named after the menu item to which you want to attach it. In my case, I name it File•Print….scpt. Notice the use of the bullet (Option+8) to separate the menu from the menu item, and the ellipsis (Option+;) to match the menu name exactly. The script should live in the “Menu Items” folder in your BBEdit Application Support folder, which you may have to create.

The script should have a menuSelect handler. If the handler returns True, then the script replaces the menu item, that is, the menu item is not run. If the handler returns false, the the menu item is run after finishing the script.

My script checks if I am trying to print a Tex document, and if so, shows an “Are you sure” dialog. If I really do want to print, the script returns False, and the print process continues.

The script is not complicated. It checks to make sure the document is a Tex document, and if so, displays a dialog. It returns True or False based on the user’s input.

on menuSelect()
    set skip_print to false
    tell application "BBEdit" to get source language of document 1
    if result is equal to "TeX" then
        try
            display dialog ¬
                "Are you sure you want to print the TeX source?" ¬
                with title "BBEdit Print" ¬
                buttons {"Don't Print", "Print"} ¬
                default button 2 ¬
                cancel button 1 ¬
                with icon caution
            get button returned of result
            set skip_print to (result is not "Print")
        on error --if they cancel the dialog
            set skip_print to true
        end try
    end if
    return skip_print
end menuSelect

You can use postmenuSelect to specify something to be run after BBEdit completes the menu item.

The moral of the story

For those of us who spend a lot of their computer time writing and manipulating text, having a good text editor and tweaking it to meet our needs can save a lot of time and frustration.

It is also very satisfying. I put in a little bit of time up front to save myself time repeatedly in the future. Then every time I use one of these little solutions, I imagine my past self saying to my current self, “Hey, let me do that for you.” This turns what would have been a tiny frustration into a tiny success.

I have been deep into Latex for the last little while, pounding out my thesis. It is a long process that involves a lot of editing, rearranging, and tweaking. Every time I revise another paragraph, I give thanks that I am using fast and powerful BBEdit, and not something slow and cumbersome like Word. But something has been missing.

A couple weeks ago, I read a post by Brandon Rhodes titled “One sentence per line”. He quotes the following advice from “UNIX for beginners” [PDF] by Brian Kernighan. Kerninghan is the K in AWK, so pay attention.

Start each sentence on a new line. Make lines short, and break lines at natural places, such as after commas and semicolons, rather than randomly. Since most people change documents by rewriting phrases and adding, deleting and rearranging sentences, these precautions simplify any editing you have to do later.

One common feature of Latex, HTML, Markdown, and many other markup languages is that they ignore single line feeds in the document source. Only my collaborators and I will be reading the source, so I can format it however I want. I might as well make life easy on myself.

Return policies

If line feeds do not affect the final document, when should I end a line?

On the one hand, I could press return only when necessary, for example, to end a paragraph. My document will contain long lines, but my text editor can wrap them for me. On the down side, the long lines could cause some trouble when I work from the command line, which generally won’t wrap lines intelligently. Also, programs which compare two versions of a file will usually be less helpful since they report differences on a line-by-line basis. (Using a word-by-word diff program will improve this, but the results are still inferior to a line-based diff of a file with short lines.)

On the other hand, I could wrap text using hard returns. Most editors will do this for me, so I’m not actually pressing return at the end of each line. This will result in shorter lines, which will make the text easier to read from the command line. In Latex I will get more precise error location, since Latex reports errors by the line on which they occur.

One disadvantage of hard wrapping is that it makes editing awkward. If I add one word to one line, I will end up pushing a single word down to the next line. Then my obsessiveness requires me to “reflow” the paragraph, removing line breaks and inserting new ones so that the lines are the same length. Even though most text editors automate the reflow process, making changes becomes quite tedious—edit, reflow, edit, reflow, edit, reflow, etc. Furthermore, each edit results in many lines changing, so comparing two documents is not much easier than when I soft wrap. Even worse, reflowing paragraphs often removes line breaks that I put in on purpose.

For example, if I make a change to the beginning of a paragraph that ends with the following text:

which can be found using the formula
\begin{equation}
    a^2 + b^2 = c^2 .
\end{equation}
This is called the Pythagorean theorem.

Reflowing is often overagressive, resulting in

which can be found using the formula \begin{equation} a^2 +
b^2 = c^2 . \end{equation} This is called the Pythagorean
theorem.

The reflowed text is still correct, and will produce the same output, but is much harder to read. This kind of thing is especially a problem in a Latex math document, where I am often switching back and forth between prose and structured mathematical formulas.

A better way

Kerninghan’s suggestion to end lines at natural breaks solves many of these issues. Documents will have short lines, which is helpful for when I need to work in a terminal or send parts of a document by email. Edits will generally change a small number of lines, which makes the diff tools work.

I get the added benefit of being able to rearrange sentences and phrases much more easily. For complicated technical statements, I can use newlines and even indentation to make the source easier to parse.

The disadvantage is that I now have to push the return key at the end of every line. For a simple document that won’t go through many revisions, this may not be worth it. But Kerninghan also points out that most documents require more revisions than we initially expect. For a dissertation, and probably any academic paper, it is worth the extra effort.

Old habits die hard

Even after a few days of ending my lines manually at natural breaks, I often find myself getting close to the “right margin”. Usually, there is a perfectly placed comma a few words back where I should have ended the line.

I made an AppleScript for BBEdit called “New line after punctuation” that looks for the last punctuation mark on the line and inserts a hard return immediately after it. I have it assigned to the keyboard shortcut Control+Return.

Update: I have made some changes the the AppleScript, explained in this post. The new script is posted here.

I know I’m extremely late to the party, but I am finally using FastScripts.

Until now, most of the scripts I use are either shell scripts that I activate from inside Terminal, or scripts that I use inside an application that supports them, like BBEdit or BibDesk.

I had tried before to find a way to run scripts from arbitrary applications, but I was never happy with what I found.

Then I found FastScripts, which of course has been around forever. It does everything I want it to. My favorite part is the ability to create application specific scripts and keyboard shortcuts, which are only shown or run when a certain application is running. That way the menu (and more importantly, the keyboard shortcut space) isn’t cluttered with irrelevant scripts.

While I’m on the subject of clutter, I recommend removing system scripts from the FastScripts menu.

Here are some scripts I collected to get my FastScripts library started.

Global

• Cancel all print jobs (Red Sweater)
• Color Picker (Ben Alman wayback link)
• Paste to BBEdit

Finder

• Duplicate window (Daniel T)
• Select files by glob (inspired by Red Sweater’s “select by regular expression”)
• Open Terminal in this directory (Red Sweater)
• Copy filename
• Calculate MD5 (inspired by RexRay)

Safari

• Read Later (Red Sweater). This uses AppleScript to run the JavaScript from the Instapaper bookmarklet.
• Add to Pinboard (same idea as above)
• Next Page (Red Sweater)
• View source in BBEdit (inspired by Ben Alman)
• Top level URL (Ben Alman wayback link)

Terminal

• Copy last output (Dr. Drang)
• Copy last input

AppleScript is a great tool. It is awesome to be able to get the selected text from an application, grab the current URL from Safari, ask the user to choose a file, or show a dialog box requesting text. But writing AppleScript scripts is usually painful.

For anything mildly complicated, I would much rather write something in Python. So a lot of my AppleScripts look like this:

1. Get information from the user or currently open application
2. do shell script some_python_or_bash_script
3. Do something with the result

For yet another time, I recently found myself making an AppleScript where part 3 of the process involved composing an email to someone. It is difficult to take the result of the shell script (which is just a single, structureless string) and parse out multiple fields (body, subject, recipient) to pass to a complicated make new message command.

So instead, I made a Python wrapper around the make new message AppleScript command. Yes, that means I am using AppleScript to call a shell script which runs an AppleScript, but I’m okay with that. (Others have done the same thing, but not with the full set of options that I wanted.)

Why use Mail.app?

There are already command line mail programs. Why not just use one of them? Two reasons.

First, getting mail to transfer properly is always a pain. Comcast won’t let you use their SMTP, and if they did, your message would probably be marked as spam. So you have to figure out how to hook authenticated SMTP up to Google, and then it breaks, and you just get sick of it. Currently, my best solution to this has been to pipe a message over SSH to my work computer, which has a fully functional transfer agent, just to send an email to myself!

Second, and more important, often you want to see the message and maybe edit it a little before you send it. This also minimizes the chance that a script will screw up and either not send the mail or send duplicates.

Create the AppleScript

AppleScript to create a mail message looks about like this:

tell application "Mail"
    make new outgoing message with properties {visible:true,¬
    subject:"Happy Birthday!",content:"The big 60!"}
    tell result
        make new to recipient with properties {address:"[email protected]"}
        make new attachment with properties {file name:"cake.jpg"}
    end tell
end tell

The first half of the Python script does nothing more than create an AppleScript and feed it to the osascript command.

#!/usr/bin/python

import sys
import argparse
import os.path
from subprocess import Popen,PIPE

def escape(s):
    """Escape backslashes and quotes to appease AppleScript"""
    s = s.replace("\\","\\\\")
    s = s.replace('"','\\"')
    return s

def make_message(content,subject=None,to_addr=None,from_addr=None,
    send=False,cc_addr=None,bcc_addr=None,attach=None):
    """Use applescript to create a mail message"""
    if send:
    properties = ["visible:false"]
    else:
    properties = ["visible:true"]
    if subject:
    properties.append('subject:"%s"' % escape(args.s))
    if from_addr:
    properties.append('sender:"%s"' % escape(args.r))
    if len(content) > 0:
    properties.append('content:"%s"' % escape(content))
    properties_string = ",".join(properties)

    template = 'make new %s with properties {%s:"%s"}'
    make_new = []
    if to_addr:
    make_new.extend([template % ("to recipient","address",
        escape(addr)) for addr in to_addr])
    if cc_addr:
    make_new.extend([template % ("cc recipient","address",
        escape(addr)) for addr in cc_addr])
    if bcc_addr:
    make_new.extend([template % ("bcc recipient","address",
        escape(addr)) for addr in bcc_addr])
    if attach:
    make_new.extend([template % ("attachment","file name",
        escape(os.path.abspath(f))) for f in attach])
    if send:
    make_new.append('send')
    if len(make_new) > 0:
    make_new_string = "tell result\n" + "\n".join(make_new) + \
        "\nend tell\n"
    else:
    make_new_string = ""

    script = """tell application "Mail"
    make new outgoing message with properties {%s}
    %s end tell
    """ % (properties_string, make_new_string)

    # run applescript
    p = Popen('/usr/bin/osascript',stdin=PIPE,stdout=PIPE)
    p.communicate(script) # send script to stdin
    return p.returncode

Dr. Drang recently complained about how inconvenient it is to send data to a subprocess in Python. I feel his pain, because I have spent plenty of time and trial and error to figure out how Popen and communicate work. The official documentation is no help, either.

In the end, though, there is nothing terribly ugly about the three lines that run the AppleScript. If you want to send anything to the subprocess’s stdin, you need the argument stdin=PIPE (or =subprocess.PIPE, depending on your import statement). Running communicate returns a tuple with the subprocess’s stdout and stderr, but only if you use the arguments stdout=PIPE and stderr=PIPE. So my script, communicate only returns the stdout (which I discard).

When you don’t specify stderr=PIPE, the error output is just passed along to the main process’s stderr (and so also with stdout). If you run my script from the command line, any errors from the osascript command will just be printed on your screen (unless, of course, you do something like 2>foo).

Use argparse

My newest rule to myself is “Never parse your own command line arguments.” Especially when I make something that I only ever plan to call from other scripts, and nobody but me is ever going to see, it is very tempting to do something stupid like require 8 positional arguments in a specific order.

Then you change some script somewhere and everything breaks. Or you want to use the script again and there is no --help. So you have to jump into source that you wrote a year ago just to figure out what to do. Not good.

The argparse library is new and replaces the short-lived and now depreciated optparse. But it has lots of useful bells and whistles. For example, with the type=argparse.FileType() option, you can add an argument that expects a filename and automatically opens the file for you. It also creates a --help option automatically.

Here is the second half of the script.

def parse_arguments():
    parser = argparse.ArgumentParser(
    description="Create a new mail message using Mail.app")
    parser.add_argument('recipient',metavar="to-addr",nargs="*",
    help="message recipient(s)")
    parser.add_argument('-s',metavar="subject",help="message subject")
    parser.add_argument('-c',metavar="addr",nargs="+",
    help="carbon copy recipient(s)")
    parser.add_argument('-b',metavar="addr",nargs="+",
    help="blind carbon copy recipient(s)")
    parser.add_argument('-r',metavar="addr",help="from address")
    parser.add_argument('-a',metavar="file",nargs="+",
    help="attachment(s)")
    parser.add_argument('--input',metavar="file",help="Input file",
    type=argparse.FileType('r'),default=sys.stdin)
    parser.add_argument('--send',action="store_true",
    help="Send the message")
    return parser.parse_args()


if __name__ == "__main__":
    args = parse_arguments()
    content = args.input.read()
    code = make_message(
        content,
        subject = args.s,
        to_addr = args.recipient,
        from_addr = args.r,
        send = args.send,
        cc_addr = args.c,
        bcc_addr = args.b,
        attach = args.a)
    sys.exit(code)

When you run parse_args, it returns a special Namespace object, which has the parsed arguments as attributes. (Why didn’t they use a dictionary?) In my script, “recipient”, which is a positional argument because it lacks a leading hyphen, is stored in args.recipient. The subject is stored in args.s. If I wanted to, I could pass ["--subject","-s"] to add_argument, and then the subject would be stored in args.subject, but could be specified on the command line as either -s subject or --subject subject. With the action="store_true" argument, args.send will be true if the user gives the --send option, and false otherwise.

I call the script mailapp. Just run

$ ls | mailapp -s "Here's how my home directory looks"

I have been using Jekyll to generate both this blog and my academic website for the past year, and I can confidently say that it has solved more problems for me than it has created. (This may sound like faint praise, but I assure you that it is not.)

Recently I have been annoyed at how long it takes to deploy updates to my website due to the way that Jekyll mangles timestamps, which rsync depends heavily on. I finally broke down and spent some time improving the process by tweaking rsync to work better with my Jekyll setup.

The Jekyll timestamp problem

It has always bothered me that Jekyll mangles timestamps. When you run jekyll to regenerate your site, all timestamps are updated to the current time. (This is because all pages are regenerated—a separate and also annoying issue.) So to anything that uses timestamps to determine when a page has changed, it appears that every page changes whenever a single page changes.

There is no solution to this problem within the Jekyll framework. Each output file is created from several input files, so you could imagine setting the timestamp of each output file to be the maximum timestamp from all of the input files. But the input files often live on several computers and/or in a git repository, which makes the timestamp of the input files both ambiguous and worthless. In these circumstances, the timestamp of a file is not the same as the last modified time of the actual data. The only way to preserve the latter is through some external database, the avoidance of which is essentially Jekyll’s raison d’être.

Rsync complications

I can overlook the fact that the file metadata on my web server is meaningless, but I have a harder time ignoring the slow deployment this causes. My academic website currently has 43 megabytes in 434 files, all but 400 kilobytes is archival stuff that never changes, and usually I am only changing a few files at a time. Nevertheless, rsync usually takes 15 seconds, even if I am transferring within the campus network.

I have two sets of files. I want to take all the differences from my local set and send them to the server set. For each pair of files, rsync checks that the sizes and modification times match, and if not, it copies the local file to the server. It has an efficient copy mechanism, so if the files are identical despite having different modification times, very little data is sent. If a large file has only changed in a few places, only the changed chunks are sent.

If you use Jekyll, the modification times never match, so all files are always copied, albeit in an efficient manner. Despite the efficient transfer mechanism, this is slow.

The correct way to use rsync with Jekyll

What you want is for rsync to compute and compare checksums for each pair of files, and only transfer files which have different checksums. You can do this by using the --checksum (or -c) option. Despite a warning from the rsync manual that “this option can be quite slow”, it reduced my transfer time from 15 seconds to 2 seconds.

Here is the command I recommend to deploy a Jekyll site:

rsync --compress \
      --recursive \
      --checksum \
      --delete \
      _site/ [email protected]:public_html/

Or, if you prefer the short version:

rsync -crz --delete _site/ [email protected]:public_html/

More meaningful timestamps on the server

A side benefit of this tweak is that server timestamps have meaning again. If the local and server files have the same checksum, nothing is copied. The timestamp of the file on the server is now the time the file was last copied to the server.

If you use the --times (or -t) option, the server timestamps are manipulated to match the (meaningless) local file timestamps. This is not what you want.

If you use the --archive (or -a) option, which is recommended by almost every rsync tutorial out there, you are implicitly using the --times option, as -a is equivalent to -rlptgoD. This is also not what you want. For a Jekyll site, the only part of -a that you care about is the -r. So don’t use -a.

Miscellaneous notes on rsync options

• The --itemize-changes (-i) option is a useful way of seeing what is transferred.
• The --ignore-times (-I) option ignores timestamps, but not in the way you want. It simply copies all files no matter what (but still using the efficient transfer mechanism).
• If you leave off the --times option and don’t use --checksum, then all files which have matching timestamps are skipped, and all other files are transferred, which changes their timestamp on the server to the current time. If you continue this over time, more and more files have different timestamps even though they are the same, which means they are copied every time.
• There is a --size-only option which skips files if they have the same size on the local computer and the server, even if they have different modification times. You are tempting fate if you use this option.

[Note: Due to changes to Heroku’s offerings, the online version of this has been shut down.]

I am teaching a differential equations course this quarter. Most of my students find slope fields very useful to visualize the set of solutions to a first order differential equation. Here is an example:

Since I know y’ as a function of y and t, I draw a short line at each point (t,y) with slope y’. Then every solution to the differential equation must run tangent to all of these little lines. If you think of the lines as the flow direction of a river, the solutions are the lines traced out by dropping a stick in the river and watching it flow.

Slope fields are incredibly easy to draw, you just have to plug in 900 or so combinations of t and y. But it isn’t something you want to do by hand. There are quite a few slope field generators out on the internet. Most of them are written as Java applets, like this one that I have pointed my students to in the past. Java applets are always painful to use and don’t work at all on the iPad, so I put together a Python script to draw the slope field.

Here are some of the different technologies that go into making this work. There is something for everyone.

Scalable Vector Graphics (SVG)

Until now, my only real experience the SVG format is the SVG gnuplot terminal, which I use to monitor the weather near my house. It is a vector format, which makes it great for line-based drawings. It is based on XML, which makes it easy to manipulate. Even better, it is human-readable. Here is the code for one tick mark:

<line x1 = "47.2" y1 = "121.5" x2 = "54.3" y2 = "127.1" />

You can imagine how easy this makes it to generate the graph using Python.

In HTML5, you can use the svg tag to embed the SVG code directly into the html file, and all current versions of major browsers support this. So if you want to see what the code looks like for the slope field above, just view this page’s source.

Python generator functions

Most of the time, I’m writing scripts to do stuff on my personal computer. I make excessive use of long lists and giant strings, never worrying about memory. This is analogous to using UNIX commands one at a time, writing the output to a temporary file each time.

Generator functions can be used to implement a pipeline structure, like when you chain UNIX commands. You can link several generator functions so that each one uses the output of another. Similar to a UNIX pipeline, the functions run in parallel, and there is no need to temporarily store the output.

Here is an example, a modified version of what I used to make the slopefield.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

def slopefield(fn,tmin,tmax,dt,ymin,ymax,dy):
    """Generator for the slopefield ticks"""
    t = tmin + 0.5 * dt
    while t < tmax:
        y = ymin + 0.5 * dy
        while y < ymax:
            # tick calculates the endpoints of the tick
            yield tick(t,y,fn)
            y += dy
        t += dt

for tick in slopefield(fn,tmin,tmax,dt,ymin,ymax,dy):
    # svg generates one line of svg code
    print svg(tick)

The yield statement makes slopefield a generator function. You could actually do this whole thing without a generator by replacing yield tick(t,y,f) with print svg(tick(t,y,f)). But when I was testing my code, I found it very useful to be able to access the intermediate data, e.g.

for tick in slopefield(fn,tmin,tmax,dt,ymin,ymax,dy):
    print tick

The great thing is that slopefield only generates its output as needed. There is no need to store it in memory somewhere.

You can only use a generator one time through.

Input Sanitization

Originally, I thought that I would parse the equations by myself, because the eval command can be used to execute arbitrary code, and that sounds pretty scary. I thought about using pyparser, but decided I was being ridiculous.

I convinced myself that it would be safe to use eval with the following sanitization, which checks every word of the string against a whitelist.

def sanitize(fn_str):
    """Sanitizes fn_str, evaluates, and returns a function"""

    VALID_WORDS = ['','sin','cos','tan','t','y','abs','sqrt','e',
        'pi','log','ln','acos','asin','atan','cosh','sinh','tanh']

    # separate into words on number and operator boundaries
    words = re.split(r'[0-9.+\-*/^ ()]+',fn_str)

    for word in words:
            if word not in VALID_WORDS:
            error('Unrecognized expression in function: %s' % word)

    s = fn_str.replace('^','**')

    # replace 1.232 with float(1.234)
    s = re.sub(r'[0-9.]+', r'float(\g<0>)', s)

    return eval("lambda t,y: " + s)

I wrap the numbers in a float command because Python does arbitrary-precision integer arithmetic, and I don’t want people to be able to type stuff like 9^9^9^9^9^9^9^9 and set the processor going crazy. That, and we avoid any integer division issues.

As noted above, I had to shut down the sample version of this due to Heroku changes, but the code is on GitHub.

Publishing a mathematics research paper is a complex and entangled process. If a math professor wants to get tenure, they need to publish their research in prestigious journals, so they submit papers to these journals. The journals, whose editors are other mathematicians, send the papers off the be peer reviewed. Neither the editors nor the reviewers are paid by the journal. The journal doesn’t even have to typeset the paper, since almost all math papers are written in Latex (99 percent, according to the sample I took of papers posted to the math section of arXiv.org). The paper is eventually published, and the journal charges a pretty high price for printed or electronic versions of the paper (sometimes over a dollar per page, which is paid primarily by university libraries). The main thing that the journal provides is its reputation, which serves as a very useful signal about the quality of the papers it contains.

There has been some talk in the past several years about open access in academia. The reasoning is that if taxpayer money is funding basic research, the results should be available to everyone at little or no cost.

The math community has open access almost completely figured out, thanks to the arXiv.org repository (which we usually refer to as “The arXiv”, which is pronounced “The Archive”, because the X stands for the greek letter chi). Although you transfer copyright to the journal when they publish your paper, what you do beforehand is generally your business. So before you submit your paper to a journal, you submit it to arXiv.org, which will post it online forever. Although I don’t know statistics, the vast majority of papers published today are listed on arXiv.org.

Journals are still necessary because of the peer review process and the prestige factor. In addition, not all authors incorporate corrections and changes that they make to their paper during the peer review process into the arXiv.org version. Actually, some journals prohibit you from doing this, although in practice, the word is if they ever tried to crack down on this there would be mass revolt.

That doesn’t mean the community is happy with journals. It would be nice if they allowed more open access, or charged less, etc. To this end, some are participating in a boycott of one company that is seen as a “worst offender”. They have comparatively high prices, they try to stop you from publishing changes to arXiv, and they lobby in favor of legislation against open access. (You probably aren’t surprised to hear they also were in favor of SOPA.)

Organizing arXiv.org papers

The reason I’ve been thinking about this is that a lot of the papers I read come from arXiv.org, and I’ve been trying to improve the way I keep track of the papers I read.

My workflow for managing papers I read goes something like this:

1. Find a paper I want to read, usually from a citation or the arXiv.org RSS feed.

2. Download a PDF to my iPad or Dropbox folder. Usually it has a stupid file name, but I don’t worry yet.

   (time passes)

3. Import the citation into BibDesk. BibDesk automatically renames the file using the author and title and moves it to my folder on my Dropbox. The difficult part is step 3. It is very useful to use BibDesk to organize the metadata for the papers I read, but it can be unnecessarily time consuming. BibDesk can import from many different websites, this usually requires you to do search using BibDesk’s built in browser for a paper that you already found once before.

This week I wrote a Python script to download arXiv.org references. It takes one or more arXiv identifiers and returns a citation for the corresponding paper in BibTeX format. Using the script I wrote, together with BibDesk’s great AppleScript integration, I can simply enter the arXiv id (which is watermarked onto the article) and everything else happens automatically.

I’m feeling a little pressure after Dr. Drang, whose blog I read regularly and benefit from often, linked to my blog. I’ve always written this blog with an eye toward future searchers of the internet, seeking to return the favor for all the unassuming blogs that have helped me stumble to a solution to my technical problems. So the fact that this blog had an average readership of 2.5 (my wife Amy being one of those) has never bothered me. And while I don’t expect a recommendation from Dr. Drang to change things too much (although I would listen to him), I feel special having suddenly gone from anonymity to having a “Gruber number” of 2 (because I was linked to by a blog that has been linked to on Daring Fireball).

From the early days, Google implemented a plus operator in its searches. When you use Google to search, they make pretty liberal use of synonyms. So if you search for “pouring”, Google also returns results for “pour” and “pours” and “poured”. If you search for “tech”, Google will also return results for “technology”. “Russian astronaut” includes pages that mention “cosmonaut”. It use to be that if you wanted to disable this behavior for a specific word, you could put a plus sign in front of it. Now that the plus sign has special (and much less useful) meaning on Google, you can get the same functionality by enclosing a single word in quotes.

But quotes are much harder than a plus sign, because they come in pairs.

My most common use scenario is this: I’m searching for something kind of obscure, and nothing useful shows up in the first page of results. Instead of clicking to the next page, I refine my search. (Honestly, if it isn’t on the first page, why would it be on the second? Or do you plan to do an exhaustive search of the index?) By looking at which words Google shows in bold, you can often tell that the synonym feature is messing with your results, and turning it off will get you what you want.

In the old days, this meant one click, one plus sign, and one press of the return key. Now, it is click, type, back to the mouse for another click, another key press, and finally return. And if I’m using my iPod or iPad? You can guarantee that at least one of those taps will require the use of the magnifying glass.

And just to prove that this is not just academic grumbling, here is one of the several times that I used this today. I used homebrew to install an updated version of imagemagick because the stock version was being dumb. While it was doing its thing, homebrew said something about “pouring” imagemagick. I headed to Google to investigate. You can imagine what results you get if you search for homebrew pouring, but even if you throw mac on to keep things relevant, you get install guides instructing you to “pour yourself a cup of coffee while you wait.” In fact, the if you narrow down your search further with a term like imagemagick, Google, which is still substituting “pour” for “pouring”, just heads to the French corner of the web, where the word “pour” is found in abundance. Some well placed quotes and a shake of my first at Google Plus for stealing my efficiency, and I’m in a much better place.

It turns out that homebrew was downloading a precompiled binary.

TikZ is a Latex package that lets you make graphics and diagrams. I have been meaning to check it out for a while now, but this week I finally had a need. Two days of class were cancelled last month due to snow, and the class I’m teaching to needs speed through some topics. So I made some notes for my students, and I drew this depiction of a linear transformation:

The advantage of TikZ over another drawing program is that it is easy to integrate it into a Latex document, matching fonts and font sizes. It is also very easy to include mathematical formulas in your drawings. And the input files are text files, which makes it easy to keep track of or share your drawings.

The advantage of TikZ over pstricks, which is another popular Latex drawing package, is that you can compile it straight to pdf, instead of generating postscript and distilling.

The code that you use to make a picture is very mathematical. This makes it a little difficult if you want to, say, make a new logo for your business, but for simple mathematical objects, it actually makes things easy. Here is an example of the code to draw one of the planes (which are, of course, actually parallelograms).

\begin{tikzpicture}
\draw[fill=blue]
    (-2, 0.6, 1.4)         % starting point
    -- ++(3.6, 0.6, 0)     % move along this vector
    -- ++(0.3, -1.8, -2.7) % then along this vector
    -- ++(-3.6, -0.6, 0)   % then back along that vector
    -- cycle; 			   % and back to where you started
\end{tikzpicture}

The minus signs tell it to make a path, and the plus signs indicate relative movement. The code for the entire picture is here.

TikZ also works great for diagrams. Having tried my hand at xypic and diagrams and amscd and probably others, I don’t think I’ll ever use anything else. If you want to see how to use TikZ for diagrams, check out this article by Felix Lenders.

Here is a BBEdit package I wrote to help me write Latex.

View on github

nathangrigg/Latex.bbpackage

This package makes it easier to use BBEdit to write Latex.

What it can do

• Typeset your document and locate any errors
• View the pdf output, with SyncTeX support if you use Skim
• View warnings and errors from the log file
• Close environments, change environments, and add or remove a star from an environment.
• Look up documentation using texdoc
• Check your semantics using chktex

There are also a number of BBEdit clippings to insert often-used Latex commands. BBEdit makes it easy to assign keyboard shortcuts to the clippings you use most.

Public key cryptography is vital to the way internet works. It was invented by mathematicians doing “pure math,” which is the kind that everyone always asks “Where in real life is that crazy math actually used?”

Here is a short introduction to how it works.

Imagine that you travel to Mars and find the planet populated by a bunch of cuddly green aliens who immediately appoint you as dictator for life. As you learn more about your newfound subjects, you find that they have an excellent knowledge of multiplication, but do not know how to do division. You make a mental note to teach them sometime.

Encryption

Meanwhile, all of your attentions are on the administration of your new kingdom. To save on collection costs, you decide to implement a system where each citizen’s taxes are withdrawn directly from his or her checking account, and you need to gather everyone’s account numbers. Of course, you want to keep things secret and secure, because identity theft has been in the news lately, and now would be a bad time for a mass revolt.

Suddenly, you get an incredible idea. You send out the following proclamation:

Please multiply your bank account number by 4 and send it to me immediately.

Sincerely, your dictator.

Everyone does so, and even though the mail system on your planet is notoriously corrupt, nobody’s account number is stolen, because you are the only one on the planet that can divide all of the numbers by 4 and thus recover the proper account numbers.

This is the first main idea of public key cryptography. You can tell everyone in the world exactly how to encrypt their data, and they all will be able to encrypt their data, but no one will be able to decrypt anybody else’s data. Only you possess that knowledge. The recipe for encrypting the data is called the public key (because everyone knows it) and the recipe for decrypting is called the private key (because only you know it).

Signatures

Some time later, someone sends out a counterfeit proclamation saying that the tax rate will be raised to 8000 martian dollars per year. An anti-tax group, thinking that the first proclamation was from you, sends out a proclamation saying that the first proclamation was a counterfeit and that the tax rate would actually be lowered to 1200 martian dollars per year. By now everyone is confused and another proclamation would probably not do much to clear anything up.

Suddenly, you get another incredible idea. You use your unique ability to divide in the opposite direction, and send the following proclamation:

The tax amount will remain at 5000 martian dollars per year. To know that I am the one that actually sent this proclamation and that the other proclamations were fake, multiply 1250 by 4 and you will see get 5000, which is the true tax rate.

Since I am the only one who knows how to divide, know one else could have possibly known that 1250 is the number that you can multiply by 4 to get the current tax rate of 5000. So you can be sure that this message is from me.

Sincerely, your dictator.

This is the second main idea of public key cryptography. You can apply the decryption process (division) to an already unencrypted message. Then anyone in the world can apply the encryption process (multiplication) and see the message. So you are not hiding anything. But since nobody else could have possibly done what you did, everyone knows that (1) the message is from you and (2) nobody has changed the message.

Because this kind of analogous to a handwritten signature certifying that you approve a message, it is called a signature. You use your private key to sign a document, and then anyone in the world can use your public key to check that it is really your signature. If someone intercepts your proclamation and changes the 5,000 to 50,000, then it will be obvious to the people that the message is no longer from you.

Certificates

This is pretty much the process that happens every time you log onto a secure website, such as your bank. Wells Fargo sends you their public key, you use it to encrypt your password, and then they are the only ones in the world who can decrypt your password. (I’m oversimplifying just a tiny bit, but this is the idea.)

There is still one problem, though. When you connect to Wells Fargo, and they send you their public key, how do you know it is really Wells Fargo, and not just someone in Nigeria pretending to be Wells Fargo?

To make sure Wells Fargo really is Wells Fargo, they don’t just send you their public key. They send you a copy of their public key that has been signed by someone your computer trusts. Your computer can check the signature of this trusted third party, which tells you that they checked that this was Wells Fargo’s key, and it hasn’t changed since then. A signed public key is usually called a certificate.

Who is the trusted third party? It is someone called a Certificate Authority (CA), whose job is to make sure that all of these public keys really belong to who they say they do. You may have heard of VeriSign, one of the most popular CA’s. As long as the CA’s are being honest (and, really, they’d be out of business otherwise), you know you are safe.

Most of this stuff happens behind the scenes, but every now and then, something goes wrong. Your web browser has probably told you “This website has an invalid certificate” at some point. What it meant was, either there are no trusted signatures on the public key, or the ones there are invalid. Usually something is just expired or set up wrong. But for all you know, someone is pretending to be something or someone they aren’t, so you should be extremely careful.

And a small amount of math

The idea of public key cryptography is pretty new. It depends on what are called one way functions which are processes that are easy to do in one direction (for example multiplication) and hard to undo (for example factoring).