Nathan Grigg

Monitor long-running commands: wait for command to finish

This is the second in a series of posts describing the system I built to monitor long-running commands.

The first post explained how I write a text file of the form $HOME/.local/logs/return-$ID every time a command finishes. The ID variable is the unique identifier for the tmux pane or iTerm window. Each file contains a single line with the return code of the last command to complete in that window.

Now that a file is being written every time a command finishes, it is not difficult to wait for a command from a given window to finish.

I call this script twait, meaning “wait for terminal.” It takes a window’s unique identifier as its only argument, waits until the corresponding return file is modified, then exits with the status read from that file.

#!/bin/bash
set -eu
returnfile="$HOME/.local/logs/return-$1"
inotifywait -qq -e close_write $returnfile && exit $(cat $returnfile)

This uses inotify (on my Linux machine at work) to wait until someone writes to the return file. Then it reads the entire contents of the return file (using cat) and uses that as the exit code for the script.

On MacOS, you can replace inotify with fswatch:

fswatch -1 $returnfile > /dev/null && exit $(cat $returnfile)

This is a suprisingly useful command. Although I originally built it with only monitoring in mind, it serves as a useful way to queue up any follow-up command. In one window I can write build, and in a separate window I can write twait %22 && ./run. This will wait for the build to finish, and as long as it is successful, run some other command.

It is not uncommon for me to have a chain of three or four tmux panes stacked vertically, each running one of a series of commands. It makes it very easy to quickly visualize the progress of the command sequence.

In the next post, I will show how I use twait to do the monitoring, and in the final post of the series I will share some tmux/iTerm integrations that make it easier to use both twait and my monitoring script.