This is the fourth and last in a series of posts describing the system I built to monitor long-running commands.

The third post explained my mon script, which together with my twait script from the second post, lights up an LED when a command in a tmux pane or iTerm window completes.

If I want to monitor a command, I use mon twait ID, where ID is the pane or window ID. Or I can run another command only if the first command is successful, by writing twait ID && ./other_command.

Of course it would be annoying to type the window ID manually, so I have two keyboard shortcuts. One to start mon twait ID in a separate window that closes automatically when it is done. The other to open a new window with the text twait ID && followed by a space.

Tmux integration

Tmux can be scripted through the use of unix-like commands, with arguments and options. These can either be run from the command line or assigned keyboard shortcuts.

I use Ctrl-A plus uppercase E for the first.

bind E run "tmux split-window -l 5 \"mon twait #D\""

This splits to form a new window, 5 lines high, and runs mon twait. The #D parameter in a tmux run command gets translated into the pane ID.

I use Ctrl-A plus lowercase E for the second.

bind e run "tmux split-window; tmux send-keys \"twait #D &&\""

iTerm integration

You can do very similar things with iTerm using AppleScript.

To open the monitor in a temporary pane, I use this script. It is possible to control the height of the new pane, but I haven’t figured it out yet.

1 tell application "iTerm"
2   tell current session of current window
3     set session_id to id
4     split horizontally with default profile ¬
5         command "mon twait " & session_id
6   end tell
7 end tell

I prefer new windows over tabs or split windows, so I use this script to open a new window prepped to run a follow-up command.

1 tell application "iTerm"
2     set session_id to id of current session of current window
3     set new_window to create window with default profile
4     set new_session to current session of new_window
5     tell new_session
6         select
7         write text "twait " & session_id & " && " without newline
8     end tell
9 end tell

This is the third in a series of posts describing the system I built to monitor long-running commands.

The second post explained my twait script, which waits for a command in a separate tmux pane or iTerm window to complete, and then exits with the same code.

The easiest way to get notified when twait finishes is to chain the commands:

twait %22; notify

But this ignores the return value, which is a useful part of the notification.

You could solve this by passing the return value as an argument:

twait %22; notify $?

The $? variable contains the return code of the last command.

I mostly use a script called mon that takes the other command as an argument:

mon twait %22

Since mon is running the command, it has access to its exit code. It can also do some prep work before running the command. The downside is that you have to keep the command fairly simple; using pipes, multiple commands, or shell aliases does not work properly.

Here is the code to my monitor script, which lights my blink LED and sends a Pushover notification.

 1 #!/bin/bash
 2 set -eu
 3 
 4 function blink() {
 5   blink1-tool -m 0 "$@" > /dev/null
 6 }
 7 
 8 blink --rgb 170,170,170
 9 echo "Monitoring $@"
10 "$@"
11 rc=$?
12 
13 case $rc in
14   0)
15     blink --green
16     title="Process succeeded"
17     ;;
18   1)
19     blink --red
20     title="Process failed"
21     ;;
22   *)
23     blink --blue
24     title="Process failed"
25     ;;
26 esac
27 
28 pushover "$title" "Exited with code $rc"

This sets the LED to a dim white at the beginning of the command, prints a message, and then runs the command. The $@ variable contains all of the arguments to the script, which should be the command to run, followed by its arguments. When we are done with the command, we capture the return code.

Depending on the value of the return code, I turn the LED green, red, or blue, and send a message to my phone and watch.

The pushover script uses curl to send a request to the pushover servers.

1 #!/bin/bash
2 set -eu
3 curl -s \
4   --form-string "token=asdfasdfasdfasdf" \
5   --form-string "user=asdfasdfasdfasdff" \
6   --form-string "title=$1" \
7   --form-string "message=$2" \
8   "https://api.pushover.net/1/messages.json" > /dev/null

Those are not my token or userid. You have to get your own from pushover.

The final post in this series will show some tmux and AppleScript integrations to make this monitoring easier to launch.

This is the second in a series of posts describing the system I built to monitor long-running commands.

The first post explained how I write a text file of the form $HOME/.local/logs/return-$ID every time a command finishes. The ID variable is the unique identifier for the tmux pane or iTerm window. Each file contains a single line with the return code of the last command to complete in that window.

Now that a file is being written every time a command finishes, it is not difficult to wait for a command from a given window to finish.

I call this script twait, meaning “wait for terminal.” It takes a window’s unique identifier as its only argument, waits until the corresponding return file is modified, then exits with the status read from that file.

#!/bin/bash
set -eu
returnfile="$HOME/.local/logs/return-$1"
inotifywait -qq -e close_write $returnfile && exit $(cat $returnfile)

This uses inotify (on my Linux machine at work) to wait until someone writes to the return file. Then it reads the entire contents of the return file (using cat) and uses that as the exit code for the script.

On MacOS, you can replace inotify with fswatch:

fswatch -1 $returnfile > /dev/null && exit $(cat $returnfile)

This is a suprisingly useful command. Although I originally built it with only monitoring in mind, it serves as a useful way to queue up any follow-up command. In one window I can write build, and in a separate window I can write twait %22 && ./run. This will wait for the build to finish, and as long as it is successful, run some other command.

It is not uncommon for me to have a chain of three or four tmux panes stacked vertically, each running one of a series of commands. It makes it very easy to quickly visualize the progress of the command sequence.

In the next post, I will show how I use twait to do the monitoring, and in the final post of the series I will share some tmux/iTerm integrations that make it easier to use both twait and my monitoring script.

This is the first in a series of posts describing the system I built to monitor long-running commands.

For me, this started as a way to light a blink 1 LED at work when a long build completed, bringing me back to the more important task from whatever may have distracted me. After a recent post by Casey Liss, I’ve added watch notifications to the mix.

The problem, for me at least, is that I never know when a command is going to take long enough that I lose interest and want a notification, so appending a command at the end of the original is not an option:

build; notify

You could always run build, and then when you realize you want notified, type notify blindly, but any stray keys or sometimes even mouse scrolling might mess up the follow up command. (Or whatever long-running command you are using might be capturing stdin.)

What I really wanted was a way for any terminal window to wait on any other terminal window. And of course, it would be nice to be able to do something different depending on what the return value is.

The first step is to capture and log the return value of every command in every window. Not long ago, this would have seemed ludicrous, but with SSDs, it really isn’t a big deal to write to disk once per command.

Unique window identifiers

To be able to tell one command from another, you need some kind of an identifier for each window. At work, I always use tmux, which calls windows panes, and gives each pane an incrementing identifier, stored in the TMUX_PANE variable. At home, I use iTerm2, which sets the TERM_SESSION_ID variable. This variable actually has two parts separated by a colon, and the unique id is in the second part, which is accessible using parameter subtitution as ${TERM_SESSION_ID#*:}.

Log the return value

Both zsh and bash can run an arbitrary command right before drawing the prompt, that is, right after any command finishes. This is a perfect place to capture and log the return value.

In zsh, this is precmd, so you can define something like this in your zshrc (using tmux pane):

precmd() {
    echo "$?" > "$HOME/.local/logs/return-$TMUX_PANE"
}

The variable $? holds the return value of the last command.

In bash, this is PROMPT_COMMAND, so you can put this in your bashrc (using iTerm window id):

PROMPT_COMMAND='echo "$?" > "$HOME/.local/logs/return-${TERM_SESSION_ID#*:}"'

In the next post, I’ll show you how to use this logged value to wait for a command to finish.